Kafka Distributed Log: How to Build Reliable Async Data Pipelines

Most developers think Kafka is just a message queue. It's not. It's a distributed log — and that distinction is the difference between a system that survives a 3AM partition failure and one that silently loses data. I've seen a payments service go down because the team treated Kafka like RabbitMQ and ran out of consumer threads. Don't be that team.

The problem Kafka solves is brutally simple: how do you reliably move data between services without tight coupling, data loss, or performance bottlenecks? Before Kafka, teams hacked together databases as message queues, polled for changes, or used point-to-point HTTP calls that failed under load. Kafka's distributed log gives you a single source of truth for event streams.

By the end of this article, you'll understand Kafka's internal architecture — the log, partitions, segments, and replication protocol — well enough to design resilient async pipelines, debug production issues, and avoid the common pitfalls that burn teams. You'll also get battle-tested code for a realistic checkout service.

The Log: Append-Only, Immutable, and Ordered

At its core, Kafka's distributed log is an append-only sequence of records. Each record has a unique offset within its partition. This immutability is what makes Kafka fast: no locks, no random writes, just sequential I/O. The log is split into segments — files on disk that are rolled over when they reach a size or time limit. Old segments are deleted or compacted based on retention policies.

Why does this matter? Because sequential disk writes are faster than random writes by orders of magnitude. Kafka can saturate a 10Gbps NIC with a single partition if the log is on fast SSDs. But here's the gotcha: if you have too many partitions (thousands), the sequential I/O advantage disappears because the OS is seeking between many small files. I've seen teams create 10,000 partitions and wonder why throughput tanked. Keep partitions per broker under 4,000 for high throughput.

// io.thecodeforge — System Design tutorial
// Simulating Kafka log segment behavior

import java.io.*;
import java.nio.file.*;

public class LogSegment {
    private final Path directory;
    private final long baseOffset;
    private Path currentSegment;
    private long currentSize;
    private final long maxSegmentBytes = 1024 * 1024 * 1024; // 1GB

    public LogSegment(Path directory, long baseOffset) {
        this.directory = directory;
        this.baseOffset = baseOffset;
        this.currentSegment = directory.resolve(baseOffset + ".log");
        this.currentSize = 0;
    }

    public void append(byte[] record) throws IOException {
        if (currentSize + record.length > maxSegmentBytes) {
            roll();
        }
        Files.write(currentSegment, record, StandardOpenOption.APPEND, StandardOpenOption.CREATE);
        currentSize += record.length;
    }

    private void roll() {
        long newOffset = baseOffset + (currentSize / 100); // simplified
        currentSegment = directory.resolve(newOffset + ".log");
        currentSize = 0;
    }

    public static void main(String[] args) throws IOException {
        LogSegment segment = new LogSegment(Paths.get("/tmp/kafka-logs"), 0);
        segment.append("order-123".getBytes());
        segment.append("order-124".getBytes());
        System.out.println("Appended records to segment: " + segment.currentSegment);
        // Output: Appended records to segment: /tmp/kafka-logs/0.log
    }
}

Partitioning: The Key to Parallelism and Ordering

Partitions are the unit of parallelism in Kafka. Each partition is an ordered, immutable log. Producers write to partitions based on a key (or round-robin if no key). Consumers read from partitions in order. This gives you a trade-off: within a partition, messages are ordered; across partitions, order is not guaranteed.

Here's the production reality: if you need global ordering, you can only have one partition. That kills throughput. Most systems don't need global ordering — they need ordering per entity (e.g., per user, per order). Use the entity ID as the partition key. I've seen teams use a random key for load balancing and then wonder why user events arrive out of order. Don't do that.

When choosing the number of partitions, consider your throughput requirements and consumer parallelism. A good rule of thumb: start with 3x the number of consumers you plan to run. You can increase partitions later, but decreasing is not supported without recreating the topic.

// io.thecodeforge — System Design tutorial
// Kafka producer with partition key for order-per-user

import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class OrderProducer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("acks", "all"); // Wait for all in-sync replicas to acknowledge

        Producer<String, String> producer = new KafkaProducer<>(props);

        // Use userId as key to ensure all events for a user go to same partition
        String userId = "user-42";
        String orderEvent = "{\"orderId\":\"order-789\",\"status\":\"created\"}";
        ProducerRecord<String, String> record = new ProducerRecord<>("orders", userId, orderEvent);

        producer.send(record, (metadata, exception) -> {
            if (exception != null) {
                System.err.println("Failed to send: " + exception.getMessage());
            } else {
                System.out.println("Sent to partition " + metadata.partition() + " at offset " + metadata.offset());
            }
        });

        producer.close();
    }
}

Replication: How Kafka Survives Broker Failures

Kafka replicates each partition across multiple brokers (configurable via replication.factor). One broker is the leader for a partition; the rest are followers. Producers write to the leader; followers replicate the data. If the leader fails, a follower becomes the new leader (controlled by the controller broker).

The key config is min.insync.replicas — the minimum number of replicas that must acknowledge a write for it to be considered committed. If you set acks=all and min.insync.replicas=2, a write must be acknowledged by the leader and at least one follower. This prevents data loss if the leader crashes after acknowledging but before the follower replicates.

Here's the gotcha: if the number of in-sync replicas drops below min.insync.replicas, the broker stops accepting writes (NotEnoughReplicasException). I've seen this happen when a broker goes down for maintenance and the team forgot to adjust min.insync.replicas. Always set min.insync.replicas to replication.factor - 1 for maximum durability, but be prepared for availability impact during broker failures.

// io.thecodeforge — System Design tutorial
// Topic creation with replication settings

// Command to create a topic with replication factor 3 and min.insync.replicas=2
// kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders \
//   --partitions 6 --replication-factor 3 --config min.insync.replicas=2

// Verify with:
// kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
// Output:
// Topic: orders	PartitionCount: 6	ReplicationFactor: 3	Configs: min.insync.replicas=2
// 	Topic: orders	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
// 	Topic: orders	Partition: 1	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1
// ...

Consumer Groups: Scaling Reads Without Losing Order

Consumer groups allow multiple consumers to read from a topic in parallel. Each partition is assigned to exactly one consumer in the group. This ensures ordering within a partition (since only one consumer reads it) while allowing horizontal scaling. If you have more consumers than partitions, some consumers will be idle.

The consumer group protocol uses a group coordinator broker and a rebalance protocol. When a consumer joins or leaves, a rebalance triggers, and partitions are reassigned. During rebalance, all consumers in the group stop processing (stop-the-world). This can cause latency spikes. I've seen teams with hundreds of consumers experience 30-second rebalances. Mitigation: use static group membership (introduced in Kafka 2.3) or cooperative rebalancing (incremental rebalance).

Another gotcha: if your consumer processes messages slowly, lag grows. But if it crashes, the group rebalances, and the new consumer starts from the last committed offset. If you use auto-commit, you might lose messages between the last commit and the crash. Always use manual offset commits after processing, not before.

// io.thecodeforge — System Design tutorial
// Consumer with manual offset commit after processing

import org.apache.kafka.clients.consumer.*;
import java.time.Duration;
import java.util.*;

public class OrderConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "order-processors");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("enable.auto.commit", "false"); // Manual commit
        props.put("max.poll.records", "100");

        Consumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Arrays.asList("orders"));

        try {
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    processOrder(record.value());
                }
                // Commit after successful batch processing
                consumer.commitSync();
            }
        } finally {
            consumer.close();
        }
    }

    private static void processOrder(String orderJson) {
        // Simulate processing (e.g., update database, call API)
        System.out.println("Processing: " + orderJson);
    }
}

Producers: Idempotence, Batching, and Throughput

Kafka producers batch records to improve throughput. The batch.size and linger.ms configs control batching. batch.size is the maximum bytes to accumulate before sending; linger.ms is the maximum time to wait for more records. Setting linger.ms=5 adds 5ms latency but can double throughput under moderate load.

Idempotent producers (enable.idempotence=true) prevent duplicate records caused by retries. When a producer sends a batch and the broker acknowledges but the producer doesn't receive the ack (e.g., network issue), the producer retries. Without idempotence, this can cause duplicate records. With idempotence, the broker deduplicates based on producer ID and sequence number. Always enable idempotence in production — the performance cost is negligible.

Here's a gotcha: if you set max.in.flight.requests.per.connection > 1 with idempotence, Kafka ensures ordering within a partition. But if you set it > 1 without idempotence, retries can cause out-of-order writes. I've seen a logging pipeline lose ordering because of this. Keep max.in.flight.requests.per.connection=5 with idempotence for high throughput.

// io.thecodeforge — System Design tutorial
// Producer with idempotence and batching

import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class HighThroughputProducer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("acks", "all");
        props.put("enable.idempotence", "true"); // Prevent duplicates
        props.put("batch.size", 16384); // 16KB
        props.put("linger.ms", 5); // Wait up to 5ms for more records
        props.put("compression.type", "snappy"); // Reduce network bandwidth

        Producer<String, String> producer = new KafkaProducer<>(props);

        for (int i = 0; i < 1000; i++) {
            String userId = "user-" + (i % 100);
            String event = "{\"eventId\":\"" + i + "\"}";
            producer.send(new ProducerRecord<>("events", userId, event));
        }

        producer.close();
    }
}

When Not to Use Kafka: The Overkill Trap

Kafka is not a message queue. If you need point-to-point messaging with low latency (sub-millisecond) and don't need replay or long-term storage, use RabbitMQ or Redis Streams. If you need a simple work queue for background jobs, use Redis or a database-backed queue. Kafka's strength is in streaming, replay, and multi-subscriber patterns.

I've seen teams use Kafka for a simple email notification service with 10 messages per second. They ended up with a 3-broker cluster, ZooKeeper (or KRaft), and a complex deployment. A single Redis instance would have handled it with less ops overhead. Don't use Kafka unless you need at least two of: replay, multiple consumer groups, or high throughput (>10K msg/s).