Event-Driven Architecture: Patterns, Brokers, and Production Strategies

Event-driven architecture is the 'social media' of microservices. In a synchronous, legacy system, Service A acts like a demanding boss—it calls Service B and waits, staring at the screen until it gets an answer. If Service B is having a bad day (or is down), Service A crashes too. In an EDA world, Service A simply 'posts' an update to a broker. It doesn't know, nor does it care, who is 'following' that update. Service B can process it now, ten minutes from now, or after it recovers from a reboot.

While this decoupling provides immense resilience, it introduces a new mental model: Eventual Consistency. You must design your system to accept that the 'User Created' email might arrive a few seconds after the database record is saved. For high-scale systems like Netflix, Uber, or LinkedIn, this trade-off is the only way to achieve global availability.

Events vs Commands vs Queries

In production system design, vocabulary is a functional constraint. Using a command when you meant an event creates accidental coupling. A Command is a 'Request for Action' (imperative), while an Event is a 'Statement of Fact' (declarative). Queries remain the synchronous backbone for immediate data retrieval.

Here's the real test: ask yourself 'Has this already happened?' If yes, it's an event. If no, it's a command. The name must reflect that. 'OrderCreated' is an event. 'CreateOrder' is a command disguised as an event. That distinction directly impacts how much coupling you ship.

package io.thecodeforge.eda.models;

import java.time.Instant;
import java.util.UUID;

/**
 * io.thecodeforge: Standardizing Intent vs Fact
 */
public class CommunicationPatterns {

    // COMMAND: Targeted intent. High coupling.
    // If the receiver is gone, the intent fails.
    public record RegisterUserCommand(
        UUID commandId,
        String email,
        String plainTextPassword
    ) {}

    // EVENT: Immutable fact. Zero coupling.
    // Multiple services (Audit, Email, Analytics) can 'sub' to this.
    public record UserRegisteredEvent(
        UUID eventId,
        UUID userId,
        String email,
        Instant occurredAt
    ) {}

    // QUERY: Immediate data request (REST/gRPC).
    public record GetUserProfileQuery(UUID userId) {}
}

The Heavyweights: Kafka vs RabbitMQ

Choosing a broker isn't about speed; it's about the 'Retention Model'. RabbitMQ is a smart post office: it routes messages to specific mailboxes and shreds the letter once it's read (acknowledged). Kafka is a distributed ledger: it appends messages to an immutable log and keeps them there for days or weeks, allowing consumers to 'rewind time' and reprocess data.

Don't fall for the raw throughput numbers. Both can handle 100k messages/sec. The real question is: can you afford to lose history? In event sourcing, you can't. Kafka's retention wins. For transient task queues, RabbitMQ's smart routing and per-queue TTL is simpler and cheaper to operate.

package io.thecodeforge.eda.kafka;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;

public class TheCodeForgeProducer {

    public void publishUserEvent(String userId, String payload) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            // Partitioning by userId ensures all events for ONE user
            // land in the same partition, preserving sequence order.
            ProducerRecord<String, String> record =
                new ProducerRecord<>("io.thecodeforge.user-events", userId, payload);

            producer.send(record, (metadata, exception) -> {
                if (exception == null) {
                    System.out.printf("Event persisted to topic %s partition %d%n",
                        metadata.topic(), metadata.partition());
                }
            });
        }
    }
}

Eventual Consistency and Trade-offs

In a synchronous system, after a write, a subsequent read immediately sees the new data. In EDA, the producer publishes an event, but the consumer may not have processed it yet. During that window, the system is inconsistent. This is eventual consistency. You must design your APIs to either return stale data or use compensating actions. For example, an order service creates an order and emits 'OrderPlaced'. The inventory service eventually decrements stock. If a read request arrives before inventory decrements, the API may report old stock levels. The trade-off: higher availability and scalability at the cost of immediate consistency.

This isn't a bug — it's a design choice. You can mitigate it with read-side caching, optimistic concurrency, or versioned responses. But you can never eliminate the inconsistency window without going synchronous.

package io.thecodeforge.eda.consistency;

public class EventualConsistencyExample {
    // On order placed, emit event: inventory decrement happens later.
    public record OrderPlacedEvent(
        String orderId,
        String userId,
        int totalAmount
    ) {}

    // The inventory service consumes and decrements asynchronously.
    // If someone queries inventory before this, they see old stock.
    // Solution: Use optimistic concurrency or version vectors.
}

Idempotent Consumer: The Non-Negotiable Pattern

Because of at-least-once delivery guarantees, the same event may arrive multiple times. An idempotent consumer ensures that processing the same event multiple times has the same effect as processing it once. The standard implementation stores processed event IDs in a database or Redis. Before processing, the consumer checks if the event ID exists. If yes, it skips the event. If no, it processes and then persists the ID. Use a unique constraint on the event ID to prevent race conditions.

In production, the race condition between 'check' and 'insert' is where duplicates creep in. Use a database INSERT with ON CONFLICT DO NOTHING or a Redis SET NX to atomically claim the event. If two consumers attempt to process the same event simultaneously, only one succeeds.

package io.thecodeforge.eda.idempotency;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;

public class IdempotentConsumer {
    public void processEvent(String eventId, Runnable processingLogic) {
        String sql = "INSERT INTO processed_events (event_id) VALUES (?) " +
                     "ON CONFLICT (event_id) DO NOTHING";
        try (Connection conn = getConnection();
             PreparedStatement stmt = conn.prepareStatement(sql)) {
            stmt.setString(1, eventId);
            int rowsInserted = stmt.executeUpdate();
            if (rowsInserted == 0) {
                System.out.println("Event " + eventId + " already processed, skipping.");
                return;
            }
        }
        processingLogic.run();
    }
}

Dead Letter Queues and Error Handling

No matter how well you test, some events will fail to process—bad data, transient dependencies, or code bugs. Instead of retrying forever (which blocks the queue), route the failed event to a Dead Letter Queue (DLQ). The DLQ holds events that exceeded retry limits. You can inspect, fix, and replay them later.

But a DLQ without monitoring is a data graveyard. Set up alerts on DLQ message count. Automate the replay pipeline: when the consumer is fixed, a script should republish DLQ events back to the main topic. Without that, events pile up silently and the system drifts.

package io.thecodeforge.eda.dlq;

import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.FixedBackOff;

// Spring Kafka: configure retries and DLQ
DefaultErrorHandler errorHandler = new DefaultErrorHandler(
    (record, exception) -> {
        System.err.println("Event failed after retries: " + record.value());
        // Send to DLQ manually or let auto-DLQ routing handle
    },
    new FixedBackOff(1000L, 3) // 3 retries with 1 sec interval
);

Schema Evolution and Compatibility

Over time, event payloads change. A producer adds a new field; consumers built six months ago need to ignore it. Schema registries (like Confluent Schema Registry) enforce compatibility rules: backward, forward, full. Backward means new consumers can read old data (default). Forward means old consumers can read new data. Full means both. Using Avro or Protobuf with a schema registry ensures safe evolution.

The #1 cause of silent event loss in production is a breaking schema change. A producer removes a required field — every consumer downstream crashes with a deserialization error, events pile up in the DLQ, and the business impact is delayed by hours. Always set defaults for new fields and enforce backward compatibility.

package io.thecodeforge.eda.schema;

import io.confluent.kafka.serializers.KafkaAvroSerializer;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.generic.GenericRecordBuilder;

// Define schema with a required field and add an optional one later
Schema userSchema = new Schema.Parser().parse(
    "{\"type\":\"record\",\"name\":\"UserCreated\",\"namespace\":\"io.thecodeforge\"," +
    "\"fields\":[{\"name\":\"id\",\"type\":\"string\"}," +
    "{\"name\":\"name\",\"type\":\"string\"}," +
    "{\"name\":\"email\",\"type\":[\"null\",\"string\"],\"default\":null}]}"
);
GenericRecord userRecord = new GenericRecordBuilder(userSchema)
    .set("id", "user-123")
    .set("name", "Alice")
    .set("email", null)
    .build();

Transactional Outbox: Preventing Dual Write Failures

A common problem: your service updates a database and then publishes an event. If the publish fails, the event is lost and other services never see the change. This is the 'Dual Write' problem. The Transactional Outbox pattern solves it: write the event into an outbox table within the same database transaction. A separate process (polling publisher or CDC) reads the outbox and publishes the event to the broker. This guarantees at-least-once publication.

For low-latency needs, use Change Data Capture (CDC) with Debezium. It reads the database transaction log and publishes events within milliseconds. No polling overhead, no risk of missing events.

package io.thecodeforge.eda.outbox;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;

public class OutboxTransaction {
    public void createOrderAndPublish(String orderId, String userId) {
        String sqlInsertOrder = "INSERT INTO orders (id, user_id) VALUES (?, ?)";
        String sqlInsertOutbox = "INSERT INTO outbox (event_type, payload, created_at) VALUES (?, ?, NOW())";

        try (Connection conn = dataSource.getConnection()) {
            conn.setAutoCommit(false);
            try (PreparedStatement orderStmt = conn.prepareStatement(sqlInsertOrder);
                 PreparedStatement outboxStmt = conn.prepareStatement(sqlInsertOutbox)) {
                orderStmt.setString(1, orderId);
                orderStmt.setString(2, userId);
                orderStmt.executeUpdate();

                outboxStmt.setString(1, "OrderCreated");
                outboxStmt.setString(2, "{\"orderId\":\"" + orderId + "\",\"userId\":\"" + userId + "\"}");
                outboxStmt.executeUpdate();

                conn.commit();
            } catch (SQLException e) {
                conn.rollback();
                throw e;
            }
        }
    }
}