Microsoft Azure — Event Hubs & Event Grid

Azure is Microsoft's cloud computing platform offering over 200 services. This article covers event hubs & event grid with production-ready configurations, best practices, and hands-on examples.

Event Hubs vs Event Grid: When to Use Which

Azure Event Hubs and Event Grid serve different purposes despite both handling events. Event Hubs is a high-throughput data streaming platform optimized for ingesting millions of events per second, with support for replay, checkpointing, and long-term retention. It's ideal for telemetry, log aggregation, and real-time analytics pipelines. Event Grid, on the other hand, is a serverless event routing service that delivers discrete events to subscribers with low latency and at-least-once delivery. It excels at reacting to state changes (e.g., blob created, VM started) and integrates natively with Azure services. The key distinction: Event Hubs is for event streaming (think Kafka), Event Grid is for event notification (think webhooks). In production, you often use both: Event Hubs ingests high-volume streams, Event Grid triggers downstream actions on those events.

# Event Hubs: high-throughput streaming
az eventhubs namespace create --name myhubns --resource-group rg --sku Standard
az eventhubs eventhub create --name telemetry --namespace-name myhubns --resource-group rg --partition-count 4 --message-retention 7

# Event Grid: reactive event routing
az eventgrid topic create --name mytopic --resource-group rg --location eastus
az eventgrid event-subscription create --name sub --source-resource-id /subscriptions/.../resourceGroups/rg/providers/Microsoft.EventGrid/topics/mytopic --endpoint https://myfunc.azurewebsites.net/api/trigger

Setting Up Event Hubs for High-Throughput Ingestion

To handle production-scale ingestion, you need to plan your Event Hubs namespace, event hub, and consumer groups carefully. Start with a Standard or Dedicated tier — Basic lacks features like auto-inflate and geo-disaster recovery. Partition count is critical: it determines the maximum parallelism for consumers. Choose a partition count based on expected throughput (each partition can handle up to 1 MB/s or 1000 events/s ingress). Use auto-inflate to automatically scale throughput units (TUs) up to your maximum. For consumer groups, create separate groups for each downstream consumer (e.g., one for real-time processing, one for archival). Avoid sharing consumer groups across different workloads — it causes checkpoint conflicts. Enable capture to automatically persist events to Azure Blob Storage or Data Lake Storage for replay and batch processing.

az eventhubs namespace create --name prodhubns --resource-group rg --sku Standard --location eastus --enable-auto-inflate --maximum-throughput-units 20
az eventhubs eventhub create --name iot-ingest --namespace-name prodhubns --resource-group rg --partition-count 16 --message-retention 7 --cleanup-policy Delete
az eventhubs eventhub consumer-group create --eventhub-name iot-ingest --namespace-name prodhubns --resource-group rg --name realtime-processor
az eventhubs eventhub consumer-group create --eventhub-name iot-ingest --namespace-name prodhubns --resource-group rg --name archival
az eventhubs eventhub consumer-group create --eventhub-name iot-ingest --namespace-name prodhubns --resource-group rg --name analytics

Producing Events with SDKs: Best Practices

When sending events to Event Hubs, use the official SDKs (C#, Java, Python, JavaScript) with batch sending and retry policies. Avoid sending one event at a time — batch at least 100 events or 1 MB per send call. Use EventHubProducerClient with a shared connection pool. Implement exponential backoff with jitter for transient failures. Set a maximum retry count (e.g., 3) and a timeout (e.g., 60 seconds). For high reliability, enable idempotent sending (Event Hubs deduplicates based on message ID). Monitor the 'Outgoing Messages' and 'Throttled Requests' metrics. If you see throttling, increase throughput units or partitions. For security, use managed identities instead of connection strings — rotate keys regularly.

from azure.eventhub import EventHubProducerClient, EventData
import asyncio

async def send_batch(producer, events):
    async with producer:
        event_data_batch = await producer.create_batch()
        for event in events:
            try:
                event_data_batch.add(EventData(event))
            except ValueError:
                await producer.send_batch(event_data_batch)
                event_data_batch = await producer.create_batch()
                event_data_batch.add(EventData(event))
        if len(event_data_batch) > 0:
            await producer.send_batch(event_data_batch)

async def main():
    conn_str = "Endpoint=sb://prodhubns.servicebus.windows.net/;SharedAccessKeyName=producer;SharedAccessKey=..."
    producer = EventHubProducerClient.from_connection_string(conn_str, eventhub_name="iot-ingest")
    events = [f"{{'id': {i}, 'temp': 22.5}}" for i in range(1000)]
    await send_batch(producer, events)

asyncio.run(main())

Consuming Events with EventProcessorClient

For reliable consumption, use EventProcessorClient (or EventHubConsumerClient for simple scenarios). EventProcessorClient manages checkpointing, partition leasing, and load balancing across multiple consumer instances. It stores checkpoints in Azure Blob Storage — each consumer group has its own checkpoint store. When scaling out, each processor instance takes ownership of a subset of partitions. Ensure the number of consumer instances does not exceed the partition count (extra instances idle). Implement error handling: if processing fails, decide whether to skip or retry. For exactly-once processing, use idempotent writes to your sink. Monitor 'Consumer Lag' metric — if it grows, add more consumer instances or optimize processing logic.

from azure.eventhub import EventHubConsumerClient
from azure.eventhub.extensions.checkpointstoreblob import BlobCheckpointStore
import asyncio

async def on_event(partition_context, event):
    print(f"Partition {partition_context.partition_id}, offset {event.offset}: {event.body_as_str()}")
    await partition_context.update_checkpoint(event)

async def main():
    conn_str = "Endpoint=sb://prodhubns.servicebus.windows.net/;SharedAccessKeyName=consumer;SharedAccessKey=..."
    checkpoint_store = BlobCheckpointStore.from_connection_string("DefaultEndpointsProtocol=https;AccountName=checkpointstore;AccountKey=...;EndpointSuffix=core.windows.net", "checkpoint-container")
    consumer = EventHubConsumerClient.from_connection_string(
        conn_str,
        consumer_group="realtime-processor",
        eventhub_name="iot-ingest",
        checkpoint_store=checkpoint_store
    )
    async with consumer:
        await consumer.receive(on_event=on_event, starting_position="-1")

asyncio.run(main())

Event Grid: Publishing Custom Events

Event Grid supports custom topics for publishing your own events. Define an event schema (CloudEvents v1.0 recommended for interoperability). Each event must have 'id', 'source', 'specversion', 'type', 'subject', 'time', 'data', and 'datacontenttype'. Use the Event Grid SDK to publish events in batches (up to 1 MB total). For high availability, enable retry policies and consider using dead-lettering for events that fail delivery. Event Grid automatically retries delivery for up to 24 hours with exponential backoff. If you need guaranteed delivery, use Event Hubs as a buffer. Monitor 'PublishFailedCount' and 'DeliveryFailedCount' metrics. For security, use SAS tokens or managed identities.

from azure.eventgrid import EventGridPublisherClient, EventGridEvent
from azure.core.credentials import AzureKeyCredential

endpoint = "https://mytopic.eastus-1.eventgrid.azure.net/api/events"
key = "..."

client = EventGridPublisherClient(endpoint, AzureKeyCredential(key))

event = EventGridEvent(
    subject="new-user",
    event_type="User.Registered",
    data={"user_id": 123, "email": "user@example.com"},
    data_version="1.0"
)

client.send([event])
print("Event published.")

Subscribing to Event Grid Events with Webhooks

Event Grid delivers events to subscribers via HTTP webhooks. Your endpoint must respond with HTTP 200 within 30 seconds, or Event Grid retries. For validation, Event Grid sends a subscription validation event with a validation code — your endpoint must echo it back. Use Azure Functions, Logic Apps, or your own webhook. For high throughput, ensure your endpoint can handle concurrent requests. Implement idempotent processing using event 'id' to deduplicate. If your endpoint is down, events are retried with exponential backoff for 24 hours, then dead-lettered. Monitor 'DeliveryFailedCount' and set alerts. For production, use Event Grid's built-in retry policy and configure a dead-letter destination.

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route('/eventgrid', methods=['POST'])
def handle_event():
    events = request.get_json()
    for event in events:
        if event['eventType'] == 'Microsoft.EventGrid.SubscriptionValidationEvent':
            return jsonify({'validationResponse': event['data']['validationCode']}), 200
        # Process your event
        print(f"Received event: {event['subject']}")
    return 'OK', 200

if __name__ == '__main__':
    app.run(port=5000)

Integrating Event Hubs with Event Grid for End-to-End Pipelines

A common pattern is to use Event Hubs for ingestion and Event Grid for downstream reactions. For example, IoT devices send telemetry to Event Hubs. A Stream Analytics job reads from Event Hubs and writes results to a database. Meanwhile, Event Grid captures events from Event Hubs (via the 'Microsoft.EventHub.CaptureFileCreated' event) to trigger archival or alerting. Alternatively, use Azure Functions with Event Hubs trigger to process events, and then publish custom events to Event Grid for further routing. This decouples ingestion from reaction. Ensure your Event Grid topic has appropriate retry and dead-letter policies. Monitor both services' metrics to detect bottlenecks.

{
  "properties": {
    "destination": {
      "endpointType": "WebHook",
      "properties": {
        "endpointUrl": "https://myfunc.azurewebsites.net/api/process"
      }
    },
    "filter": {
      "subjectBeginsWith": "/blobServices/default/containers/telemetry/",
      "includedEventTypes": ["Microsoft.EventHub.CaptureFileCreated"]
    },
    "retryPolicy": {
      "maxDeliveryAttempts": 10,
      "eventTimeToLiveInMinutes": 1440
    },
    "deadLetterDestination": {
      "endpointType": "StorageBlob",
      "properties": {
        "resourceId": "/subscriptions/.../resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/deadletter",
        "blobContainerName": "deadletter"
      }
    }
  }
}

Monitoring and Alerting for Event Hubs and Event Grid

Production systems require proactive monitoring. For Event Hubs, key metrics: Incoming Messages, Outgoing Messages, Throttled Requests, Consumer Lag, and Active Connections. Set alerts on Throttled Requests > 0 (indicates need to scale) and Consumer Lag > 1000 (consumers falling behind). For Event Grid, monitor PublishFailedCount, DeliveryFailedCount, and DeadLetteredCount. Use Azure Monitor and Log Analytics to correlate events. Enable diagnostic settings to stream logs to a Log Analytics workspace. For Event Hubs, capture logs for audit. For Event Grid, log delivery failures. Create dashboards for real-time visibility. Use Azure Alerts with action groups to notify on-call engineers.

# Event Hubs alert for throttled requests
az monitor metrics alert create --name "EventHubs Throttling" --resource-group rg --scopes /subscriptions/.../resourceGroups/rg/providers/Microsoft.EventHub/namespaces/prodhubns --condition "ThrottledRequests > 0" --description "Alert when throttling occurs" --action /subscriptions/.../resourceGroups/rg/providers/microsoft.insights/actiongroups/myag

# Event Grid alert for delivery failures
az monitor metrics alert create --name "EventGrid Delivery Failures" --resource-group rg --scopes /subscriptions/.../resourceGroups/rg/providers/Microsoft.EventGrid/topics/mytopic --condition "DeliveryFailedCount > 0" --description "Alert on delivery failures" --action /subscriptions/.../resourceGroups/rg/providers/microsoft.insights/actiongroups/myag

Disaster Recovery and Geo-Disaster for Event Hubs

For business continuity, Event Hubs offers Geo-disaster recovery (paired region) and Availability Zones. Geo-disaster recovery replicates the namespace configuration (not events) to a secondary region. On failover, you manually switch to the secondary namespace. For active-active replication, use Event Hubs' Capture to copy events to a geo-replicated storage account, or use a custom replication solution. Availability Zones protect within a region. For Event Grid, it's region-resilient by default (Microsoft-managed). For custom topics, you can publish to multiple regions and use Traffic Manager. Test failover regularly. Document runbooks for manual failover steps.

# Create primary and secondary namespaces
az eventhubs namespace create --name primaryhub --resource-group rg --location eastus --sku Standard
az eventhubs namespace create --name secondaryhub --resource-group rg --location westus --sku Standard

# Create geo-recovery alias
az eventhubs georecovery-alias create --name myalias --resource-group rg --namespace-name primaryhub --partner-namespace secondaryhub

# Pair the namespaces
az eventhubs georecovery-alias set --name myalias --resource-group rg --namespace-name primaryhub --partner-namespace secondaryhub

Cost Optimization for Event Hubs and Event Grid

Event Hubs pricing is based on throughput units (TUs) or processing units (PUs) for Dedicated. For Standard, you pay per TU per hour plus ingress/egress. Use auto-inflate to scale TUs based on load, but set a maximum to control costs. For low-throughput scenarios, consider using Basic tier (but limited features). For Event Grid, you pay per million operations (publish, delivery, etc.). To reduce costs, batch events, filter subscriptions to only relevant event types, and use dead-lettering sparingly. Monitor cost in Azure Cost Management. For Event Hubs, consider using Dedicated cluster if you have predictable high throughput (reserved capacity discount).

# Estimate Event Hubs cost (Standard, 1 TU, 1 million ingress operations/month)
# Pricing: $0.015 per TU hour, $0.028 per million ingress operations
# Monthly cost = (1 TU * 730 hours * $0.015) + (1 million * $0.028) = $10.95 + $0.028 = $10.978

# Estimate Event Grid cost (1 million operations)
# Pricing: $0.60 per million operations (first 100k free)
# Monthly cost = $0.60

Security Best Practices for Event Hubs and Event Grid

Use managed identities instead of connection strings for authentication. For Event Hubs, assign roles like 'Azure Event Hubs Data Sender' and 'Azure Event Hubs Data Receiver' to identities. For Event Grid, use 'EventGrid Data Sender' for publishers. Enable network security: use private endpoints to restrict access to your virtual network, and disable public network access. For Event Hubs, enable firewall rules to allow only trusted IPs. Use Azure Policy to enforce minimum TLS version (1.2). Rotate SAS keys regularly if you must use them. Enable diagnostic logs for audit. For sensitive data, encrypt events at rest (default) and consider customer-managed keys (CMK) for compliance.

# Assign managed identity to VM and grant Event Hubs sender role
az vm identity assign --name myvm --resource-group rg --identities /subscriptions/.../resourcegroups/rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/myid
az role assignment create --assignee-object-id $(az vm show --name myvm --resource-group rg --query identity.principalId -o tsv) --role "Azure Event Hubs Data Sender" --scope /subscriptions/.../resourceGroups/rg/providers/Microsoft.EventHub/namespaces/prodhubns

# Enable private endpoint for Event Hubs
az network private-endpoint create --name mype --resource-group rg --vnet-name myvnet --subnet mysubnet --private-connection-resource-id /subscriptions/.../resourceGroups/rg/providers/Microsoft.EventHub/namespaces/prodhubns --group-id namespace --connection-name myconnection

Troubleshooting Common Issues in Production

Common issues: throttling (increase TUs/partitions), consumer lag (scale consumers, optimize processing), checkpoint failures (ensure blob storage is accessible), Event Grid delivery failures (check endpoint availability, dead-letter config). For Event Hubs, check if the namespace is in 'Activating' state (wait). For Event Grid, validate webhook endpoint responds within 30s. Use Azure Monitor to correlate errors. For Event Hubs, enable diagnostic logs and look for 'QuotaExceeded' or 'Timeout' errors. For Event Grid, check 'DeliveryFailedCount' and dead-letter storage. Have a runbook for each issue. Test failover scenarios regularly.

# Check Event Hubs metrics
az monitor metrics list --resource /subscriptions/.../resourceGroups/rg/providers/Microsoft.EventHub/namespaces/prodhubns --metric "ThrottledRequests" --interval PT1M

# Check Event Grid delivery failures
az monitor metrics list --resource /subscriptions/.../resourceGroups/rg/providers/Microsoft.EventGrid/topics/mytopic --metric "DeliveryFailedCount" --interval PT1M

# View Event Hubs diagnostic logs
az monitor diagnostic-settings list --resource /subscriptions/.../resourceGroups/rg/providers/Microsoft.EventHub/namespaces/prodhubns

Kafka Compatibility in Event Hubs: AMQP vs Kafka Protocol

Azure Event Hubs natively supports the Apache Kafka protocol alongside its native AMQP protocol. This means existing Kafka applications can publish and consume from Event Hubs with zero code changes — just update the bootstrap server to your Event Hubs namespace endpoint. The Kafka protocol support includes Kafka producers, consumers, Kafka Connect, Kafka Streams, and MirrorMaker. Key differences from Apache Kafka: Event Hubs doesn't support Kafka's native topic compaction (though log compaction is available as a preview feature), and it uses a flat namespace (no ZooKeeper). For Kafka users, Event Hubs provides a managed, auto-scaling alternative without the operational overhead of managing Kafka clusters. However, there are limitations: transactional writes, idempotent producers, and some consumer group management features work differently. Use the AMQP SDK for maximum Azure integration (e.g., managed identity, schema registry) and Kafka protocol for existing Kafka workloads or multi-cloud portability.

# Kafka producer configured for Event Hubs
bootstrap.servers=myhubns.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://myhubns.servicebus.windows.net/;SharedAccessKeyName=producer;SharedAccessKey=...";

# Producer config
acks=all
retries=3
enable.idempotence=false  # Not fully supported; use at-least-once

# Consumer config
group.id=my-consumer-group
auto.offset.reset=earliest
enable.auto.commit=false

Schema Registry: Managing Event Schemas with Avro and JSON

Azure Schema Registry, integrated with Event Hubs, provides centralized schema management for event streaming. It supports Avro, JSON Schema, and Protobuf formats. Producers register schemas and serialize events with a schema ID; consumers deserialize using the registry. This ensures data compatibility across services — a schema change doesn't break downstream consumers. Schema groups organize schemas by business domain. Compatibility modes (Backward, Forward, Full, None) enforce evolution rules: Backward allows deleting fields, Forward allows adding fields. Use the azure-schema-registry-avro library for .NET, Java, Python, and JavaScript integration. For Kafka applications, the azure-schema-registry-for-kafka library provides Avro serializers. Key best practice: always validate schema changes against the compatibility mode in CI/CD before deploying new producers. Without Schema Registry, producers and consumers must agree on schemas out-of-band, leading to deserialization failures in production.

using Azure.Data.SchemaRegistry;
using Azure.Identity;
using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Producer;
using Avro.Generic;

// Create Schema Registry client
var schemaRegistry = new SchemaRegistryClient(
    "<namespace>.servicebus.windows.net",
    new DefaultAzureCredential());

// Register schema
var schema = "{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}";
var schemaProps = await schemaRegistry.RegisterSchemaAsync(
    "orders-sg", "Order", SchemaFormat.Avro, schema);

// Send event with schema reference
var producer = new EventHubProducerClient(
    "<connection-string>", "orders");
using var batch = await producer.CreateBatchAsync();
batch.TryAdd(new EventData(BinaryData.FromString(schemaProps.Value.Id)));
await producer.SendAsync(batch);

Event Grid Domains and Advanced Filtering

Event Grid Domains provide a management endpoint for publishing to multiple topics within a single domain. Each topic is isolated, with its own subscriptions and access control, but shares a common publish endpoint. This is ideal for multi-tenant eventing: each tenant gets a topic, and the domain routes events based on the 'topic' field. Domain topics support advanced filtering with subject matching (BeginsWith, EndsWith), event type filtering, and extended properties filtering. For complex routing, use Event Grid's dead-letter destinations with Storage Blobs to capture undelivered events. A common pattern: use domains for SaaS platforms where each customer's events are isolated. Monitor domain-level metrics with 'PublishSuccessCount' and 'DeliveryFailedCount'. Domains do not limit throughput like regular topics (5k events/s) — scale is per domain endpoint. However, be aware that domain pricing includes a per-event surcharge.

# Create Event Grid domain
az eventgrid domain create \
  --name myeventdomain \
  --resource-group rg \
  --location eastus \
  --sku basic

# Create topic within domain
az eventgrid domain topic create \
  --domain-name myeventdomain \
  --resource-group rg \
  --name tenant-orders

# Subscribe with advanced filter
az eventgrid event-subscription create \
  --source-resource-id /subscriptions/.../resourceGroups/rg/providers/Microsoft.EventGrid/domains/myeventdomain/topics/tenant-orders \
  --name billing-sub \
  --endpoint https://billing.azurewebsites.net/api/events \
  --advanced-filter data.orderTotal NumberGreaterThan 1000 \
  --deadletter-endpoint /subscriptions/.../resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/dlq/blobServices/default/containers/deadletter