Amazon EventBridge: Event-Driven Architecture at Scale

You’ve got 50 microservices, each emitting events. Your first attempt was SNS-to-SQS fan-out. Then came the spaghetti of filters, dead-letter queues, and manual retry logic. One misconfigured subscription took down your entire order pipeline for 45 minutes. That’s when you realize: point-to-point event routing is a liability at scale.

EventBridge solves this by acting as a centralized event bus with schema discovery, content-based filtering, and built-in retry policies. It’s not just a fan-out mechanism—it’s a control plane for your events. You can route the same event to multiple targets, transform payloads, and archive events for replay. And because it’s serverless, you don’t think about partitions, throughput, or scaling—EventBridge handles it.

But here’s the catch: EventBridge’s default retry policy is naive. If your Lambda fails, EventBridge retries up to 24 hours by default—potentially flooding your DLQ with stale events. You need to configure retry policies, dead-letter queues, and idempotency from day one. This article walks through production patterns that prevent the ‘EventBridge surprise’—the moment your bus becomes a firehose of failures.

Why Event-Driven Architecture Matters at Scale

Monolithic synchronous architectures break under load. When every service calls every other service via HTTP, you get tight coupling, cascading failures, and a debugging nightmare. Event-driven architecture (EDA) decouples producers from consumers using an event bus. Amazon EventBridge is AWS's fully managed event bus that scales to millions of events per second. It handles schema discovery, filtering, and routing so you can focus on business logic. At TheCodeForge, we've seen teams reduce pager duty alerts by 70% after migrating to EventBridge. The key insight: events are facts, not commands. A producer emits an event like "OrderPlaced" and doesn't care who listens. Consumers react asynchronously. This pattern enables independent deployments, fault isolation, and natural load leveling. But EDA introduces complexity: event ordering, idempotency, and observability. EventBridge addresses some of these with features like replay archives and schema registries. Start with a simple use case: emit events for critical domain actions (order placed, payment received, user registered). Avoid the trap of making every database change an event—that leads to event spaghetti. Instead, model events around business outcomes.

import { EventBridgeClient, PutEventsCommand } from "@aws-sdk/client-eventbridge";

const client = new EventBridgeClient({ region: "us-east-1" });

export async function emitOrderPlaced(order: { id: string; total: number; userId: string }) {
  const command = new PutEventsCommand({
    Entries: [
      {
        EventBusName: "default",
        Source: "com.thecodeforge.orders",
        DetailType: "OrderPlaced",
        Detail: JSON.stringify(order),
        Resources: [order.id],
        Time: new Date(),
      },
    ],
  });

  const response = await client.send(command);
  if (response.FailedEntryCount && response.FailedEntryCount > 0) {
    console.error("Failed entries:", response.Entries?.filter(e => e.ErrorCode));
    throw new Error("Event publish failed");
  }
  return response;
}

EventBridge Architecture: Buses, Rules, and Targets

EventBridge has three core components: event buses, rules, and targets. An event bus is a pipeline that receives events. The default bus receives AWS service events (e.g., EC2 state changes, CloudTrail API calls). You can create custom buses for your application events. Rules define which events to match and where to send them. Rules can filter events by source, detail type, or content using JSONPath expressions. Targets are the destinations: Lambda functions, Step Functions, SQS queues, SNS topics, API Gateway, or even other AWS accounts. You can also archive events for replay. The magic is in the filtering: you can route events to different targets based on event content without writing any code. For example, route "OrderPlaced" events with total > $1000 to a fraud detection Lambda, and others to a fulfillment queue. This reduces Lambda invocations and costs. EventBridge also supports input transformers to reshape the event before sending to the target. This is critical when integrating with legacy systems that expect a different payload. One gotcha: EventBridge has a 256 KB payload limit per event. For larger payloads, store the data in S3 and include a reference in the event.

resource "aws_cloudwatch_event_rule" "high_value_order" {
  name           = "high-value-order-rule"
  event_bus_name = "default"
  event_pattern  = jsonencode({
    source      = ["com.thecodeforge.orders"]
    detail-type = ["OrderPlaced"]
    detail = {
      total = [{ numeric = [">", 1000] }]
    }
  })
}

resource "aws_cloudwatch_event_target" "fraud_detection" {
  rule           = aws_cloudwatch_event_rule.high_value_order.name
  event_bus_name = "default"
  arn            = aws_lambda_function.fraud_detection.arn
  input_transformer {
    input_paths = {
      order_id = "$.detail.id",
      total    = "$.detail.total"
    }
    input_template = <<EOF
{
  "orderId": <order_id>,
  "amount": <total>,
  "source": "eventbridge"
}
EOF
  }
}

Schema Discovery and Code Generation

EventBridge includes a schema registry that automatically discovers event schemas as they pass through the bus. This is a game-changer for team collaboration. When a producer emits an event, EventBridge infers the schema (Avro format) and stores it. You can then generate strongly-typed client libraries for Java, Python, TypeScript, and more. This eliminates manual schema documentation and reduces integration bugs. To enable schema discovery, you must turn it on per event bus (it's off by default). Once enabled, you can view schemas in the console or download them via API. The generated code includes classes for events and serialization helpers. For example, a TypeScript client will have an OrderPlaced interface with typed fields. This catches errors at compile time rather than runtime. However, schema discovery has a cost: $0.001 per 1000 events scanned. For high-throughput buses, this can add up. Also, schemas are versioned—if you change an event structure, a new version is created. You can set a schema registry as the source of truth for event contracts. But beware: schema discovery only works for JSON events. If you send non-JSON events, they are ignored. Also, events with dynamic keys (e.g., map of user IDs) can produce overly complex schemas. In practice, we recommend enabling schema discovery for development and staging environments, but consider disabling it for production if costs are a concern.

// Auto-generated from EventBridge schema registry
// Schema: com.thecodeforge.orders@OrderPlaced (version 1)

export interface OrderPlaced {
  id: string;
  total: number;
  userId: string;
  items: Array<{
    productId: string;
    quantity: number;
    price: number;
  }>;
  timestamp: string; // ISO 8601
}

// Usage in consumer
export const handler = async (event: OrderPlaced) => {
  console.log(`Processing order ${event.id} for user ${event.userId}`);
  // ...
};

Event Replay and Archive for Debugging

One of EventBridge's most powerful features is event archiving and replay. You can configure an archive on an event bus to store all events for a specified retention period (1-365 days). Then, you can replay events from a specific time range to a target. This is invaluable for debugging, testing, and backfilling. For example, if a bug in your consumer caused events to be processed incorrectly, you can fix the consumer and replay the events from the archive. Archives are stored in EventBridge's internal storage, not S3, so you don't manage lifecycle. However, archives have a cost: $0.01 per GB per month for storage, plus $0.001 per 1000 events replayed. For high-volume events, costs can add up. Also, archives capture events after filtering? No, archives capture all events that pass through the bus, regardless of rules. So if you have a rule that filters out 90% of events, the archive still stores all of them. To reduce costs, consider archiving only specific event buses or using a separate bus for archival. Replay is not instantaneous—it can take minutes to hours depending on the volume. Also, replay events are re-ingested into the bus, so they will trigger rules again. This can cause duplicate processing if your consumer is not idempotent. Always design consumers to handle duplicates (e.g., using idempotency keys). Another use case: replay events into a staging environment to test new consumer versions against real production data.

# Create an archive on the default bus (retention 30 days)
aws events create-archive \
  --archive-name order-events-archive \
  --event-source-arn arn:aws:events:us-east-1:123456789012:event-bus/default \
  --retention-days 30

# Replay events from a specific time range to a Lambda function
aws events start-replay \
  --replay-name replay-20250301 \
  --event-source-arn arn:aws:events:us-east-1:123456789012:event-bus/default \
  --destination Arn=arn:aws:lambda:us-east-1:123456789012:function:replay-consumer \
  --event-start-time 2025-03-01T00:00:00Z \
  --event-end-time 2025-03-01T23:59:59Z

Cross-Account Event Routing with EventBridge

In multi-account AWS organizations, you often need to route events between accounts. EventBridge supports cross-account event buses. You can create a rule on a source account's bus that sends events to a target bus in another account. This is useful for centralizing events (e.g., all security events to a central audit account) or for sharing domain events across microservices owned by different teams. To set up cross-account routing, you need to: 1) In the source account, create a rule with a target of the destination account's event bus ARN. 2) In the destination account, create a resource-based policy on the event bus that allows the source account to put events. 3) Optionally, create a rule in the destination account to route incoming events to local targets. Cross-account events are encrypted in transit and at rest. However, there are limits: you can only send events to a bus in the same region (cross-region is not supported natively; you'd need to use a Lambda function to forward). Also, cross-account events count against the source account's event throughput. If the destination account's bus is throttled, events are retried for up to 24 hours. One common pattern is to use a central event bus in a shared services account. Each application account sends domain events to the central bus, which then routes to various consumers (e.g., data lake, monitoring). This reduces coupling between teams. But be careful: cross-account event delivery adds latency (typically < 1 second, but can be higher). Also, debugging cross-account issues requires CloudTrail in both accounts.

# Source account (application account)
resource "aws_cloudwatch_event_rule" "send_to_central" {
  name           = "send-to-central-bus"
  event_bus_name = "default"
  event_pattern  = jsonencode({
    source = ["com.thecodeforge.orders"]
  })
}

resource "aws_cloudwatch_event_target" "central_bus" {
  rule           = aws_cloudwatch_event_rule.send_to_central.name
  event_bus_name = "default"
  arn            = "arn:aws:events:us-east-1:999999999999:event-bus/central-bus"
  role_arn       = aws_iam_role.eventbridge_cross_account.arn
}

# Destination account (central account)
resource "aws_cloudwatch_event_bus_policy" "allow_source" {
  event_bus_name = "central-bus"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowSourceAccount"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::123456789012:root"
        }
        Action   = "events:PutEvents"
        Resource = "arn:aws:events:us-east-1:999999999999:event-bus/central-bus"
      }
    ]
  })
}

Error Handling and Dead Letter Queues

EventBridge retries failed deliveries to targets for up to 24 hours with exponential backoff. But if a target is consistently failing (e.g., a Lambda function that throws an exception), events will be retried and eventually dropped. To prevent data loss, configure a dead letter queue (DLQ) on the target. The DLQ can be an SQS queue or an SNS topic. When EventBridge exhausts retries, it sends the event to the DLQ. You can then process DLQ events manually or with a separate consumer. DLQs are essential for production systems. Without them, you lose events silently. Configure DLQs on all critical targets. Also, consider setting a retry policy: maximum retry count (default 185) and maximum event age (default 24 hours). For time-sensitive events, reduce the retry count. For example, a payment confirmation event should be retried only a few times because the user expects a quick response. Another pattern: use a Lambda function to process DLQ events, log them, and alert the team. You can also replay DLQ events back to the original target after fixing the issue. However, DLQ events are not automatically replayed—you need to manually process them. One gotcha: if the target is an SQS queue, the DLQ must be in the same region. Also, DLQ events include the original event plus metadata (e.g., error message). Use this metadata to debug failures.

resource "aws_sqs_queue" "event_dlq" {
  name = "order-events-dlq"
}

resource "aws_cloudwatch_event_target" "order_processor" {
  rule           = aws_cloudwatch_event_rule.order_placed.name
  event_bus_name = "default"
  arn            = aws_lambda_function.order_processor.arn
  dead_letter_config {
    arn = aws_sqs_queue.event_dlq.arn
  }
  retry_policy {
    maximum_retry_attempts       = 5
    maximum_event_age_in_seconds = 300
  }
}

# Lambda to process DLQ
resource "aws_lambda_function" "dlq_processor" {
  filename         = "dlq_processor.zip"
  function_name    = "dlq-processor"
  role             = aws_iam_role.lambda_role.arn
  handler          = "index.handler"
  runtime          = "nodejs18.x"
  source_code_hash = filebase64sha256("dlq_processor.zip")
  environment {
    variables = {
      DLQ_URL = aws_sqs_queue.event_dlq.id
    }
  }
}

Monitoring and Observability with EventBridge

EventBridge emits CloudWatch metrics for each event bus: number of events, failed invocations, throttled events, and more. You can set up dashboards and alarms to monitor event flow. Key metrics to watch: Invocations (events delivered), FailedInvocations (delivery failures), ThrottledRules (rules that exceeded target concurrency), and MatchedEvents (events that matched at least one rule). A sudden drop in MatchedEvents could indicate a broken rule or a producer issue. Also, monitor EventAge to detect processing delays. For detailed debugging, enable CloudTrail data events for EventBridge. This logs every PutEvents call, including the event payload. However, CloudTrail can be expensive for high-throughput buses. Instead, use EventBridge's built-in archive and replay for debugging. Another powerful tool is EventBridge's schema registry, which shows you the structure of events flowing through the bus. You can also use AWS X-Ray to trace events through targets, but X-Ray integration is limited to Lambda and API Gateway targets. For custom targets (e.g., SQS), you need to add tracing manually. One common pattern: emit a custom metric from your consumer Lambda (e.g., OrderProcessedSuccessfully) to track business KPIs. This gives you end-to-end visibility. Also, log the event ID and target response in your consumer for correlation. Finally, set up a CloudWatch dashboard that shows event bus metrics, DLQ depth, and consumer health.

import { CloudWatchClient, PutMetricDataCommand } from "@aws-sdk/client-cloudwatch";

const cloudwatch = new CloudWatchClient({ region: "us-east-1" });

export async function recordOrderProcessed(orderId: string, status: string) {
  const command = new PutMetricDataCommand({
    Namespace: "TheCodeForge/Orders",
    MetricData: [
      {
        MetricName: "OrderProcessed",
        Value: 1,
        Unit: "Count",
        Dimensions: [
          { Name: "Status", Value: status },
        ],
        Timestamp: new Date(),
      },
    ],
  });
  await cloudwatch.send(command);
}

// In consumer Lambda
export const handler = async (event: any) => {
  try {
    // process order
    await recordOrderProcessed(event.detail.id, "Success");
  } catch (err) {
    await recordOrderProcessed(event.detail.id, "Failure");
    throw err;
  }
};

Performance Tuning and Throttling

EventBridge scales automatically, but there are limits. By default, each event bus can handle up to 10,000 events per second (can be increased via service quota). Targets also have concurrency limits. For example, a Lambda function has a reserved concurrency limit. If the target is throttled, EventBridge retries. But if the target is consistently throttled, events will back up and eventually be dropped. To avoid this, use SQS as a buffer between EventBridge and your consumer. SQS can absorb spikes and decouple the consumer's processing rate from the event arrival rate. Configure a Lambda function to poll the SQS queue with batch processing. This pattern is called "SQS as a buffer." Another performance consideration: event payload size. The maximum is 256 KB. If your events are larger, store the payload in S3 and include a reference. Also, avoid sending many small events individually—batch them using PutEvents with multiple entries (up to 10 per call). This reduces API calls and costs. EventBridge also has a limit of 300 rules per event bus. If you need more, create additional custom buses. For high-throughput scenarios, consider partitioning events across multiple buses (e.g., by region or tenant). Finally, monitor ThrottledRules metric. If you see throttling, increase the target's concurrency or add a buffer.

import { EventBridgeClient, PutEventsCommand } from "@aws-sdk/client-eventbridge";

const client = new EventBridgeClient({ region: "us-east-1" });

export async function emitBatchOrders(orders: Array<{ id: string; total: number; userId: string }>) {
  const entries = orders.map(order => ({
    EventBusName: "default",
    Source: "com.thecodeforge.orders",
    DetailType: "OrderPlaced",
    Detail: JSON.stringify(order),
    Time: new Date(),
  }));

  // Split into batches of 10 (max per PutEvents)
  for (let i = 0; i < entries.length; i += 10) {
    const batch = entries.slice(i, i + 10);
    const command = new PutEventsCommand({ Entries: batch });
    const response = await client.send(command);
    if (response.FailedEntryCount && response.FailedEntryCount > 0) {
      console.error("Failed entries in batch:", response.Entries?.filter(e => e.ErrorCode));
    }
  }
}

Security: IAM Policies and Encryption

EventBridge integrates with AWS IAM for access control. You need permissions to put events to a bus and to allow rules to invoke targets. For cross-account, use resource-based policies. For encryption, EventBridge encrypts events at rest using AWS KMS (default AWS managed key or customer managed key). You can also enforce encryption in transit via HTTPS. When configuring targets, ensure the target's resource policy allows EventBridge to invoke it. For example, a Lambda function's resource policy must grant lambda:InvokeFunction to EventBridge. Similarly, SQS queues need a policy that allows EventBridge to send messages. One common mistake: forgetting to attach the correct IAM role to the rule when the target is in a different account. The role must have permissions to put events to the target bus. Also, use least privilege: restrict the source and detail type in the event pattern to limit what events the rule can match. For sensitive events, consider using a custom event bus with a strict resource policy. Finally, enable CloudTrail to audit all PutEvents calls. This helps with compliance and incident investigation.

resource "aws_iam_role" "eventbridge_invoke_lambda" {
  name = "eventbridge-invoke-lambda-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "events.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy" "invoke_lambda" {
  name = "invoke-lambda-policy"
  role = aws_iam_role.eventbridge_invoke_lambda.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = "lambda:InvokeFunction"
        Resource = aws_lambda_function.order_processor.arn
      }
    ]
  })
}

# Lambda resource policy
resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.order_processor.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.order_placed.arn
}

Cost Optimization Strategies

EventBridge pricing is based on events published ($1.00 per million events) and rules evaluated ($1.00 per million rule evaluations). For high-throughput systems, costs can escalate. Here are strategies to optimize: 1) Batch events: Use PutEvents with multiple entries (up to 10) to reduce the number of API calls. Each entry counts as one event, but you pay per event, not per API call. 2) Filter early: Use event patterns to match only relevant events. Unmatched events are still counted as events, but they don't incur rule evaluation costs. However, you still pay for the event itself. 3) Use custom event buses: Separate high-volume events from low-volume ones. You can have a bus for critical business events and another for logging events. This makes cost tracking easier. 4) Archive selectively: Only archive events that you might need to replay. Archives cost $0.01 per GB per month. For high-volume events, consider archiving only a sample (e.g., 1% of events) or use S3 for long-term storage. 5) Monitor costs: Use AWS Cost Explorer to track EventBridge costs by bus and source. Set budgets and alerts. 6) Consider SNS for simple fan-out: If you don't need content-based filtering or schema discovery, SNS is cheaper ($0.50 per million deliveries). 7) Use Lambda reserved concurrency to avoid throttling, which can cause retries and increase costs. 8) Delete unused rules and buses. Old rules still incur evaluation costs if they match events.

# Use a custom event bus for high-volume events to isolate costs
resource "aws_cloudwatch_event_bus" "high_volume" {
  name = "high-volume-bus"
}

# Archive only critical events (e.g., orders) not all events
resource "aws_cloudwatch_event_archive" "critical_orders" {
  archive_name     = "critical-orders-archive"
  event_source_arn = aws_cloudwatch_event_bus.high_volume.arn
  retention_days   = 7
  # Use event pattern to archive only OrderPlaced events
  event_pattern    = jsonencode({
    detail-type = ["OrderPlaced"]
  })
}