Amazon Kinesis: Real-Time Data Streaming and Analytics – Production Patterns That Don't Fall Over

You're running a real-time analytics pipeline. Data flows in from thousands of devices, and you need to process it, enrich it, and land it in S3 within seconds. You set up Kinesis, write a Lambda consumer, and it works in dev. Then production hits. Your Lambda starts throttling, shards are hot, and your bill explodes. Sound familiar? Kinesis is deceptively simple, but the defaults will burn you at scale.

The problem Kinesis solves is ingesting and processing high-throughput streaming data without building your own Kafka cluster. It handles durability, replication, and automatic failover. But the trade-off is a fixed shard model that requires careful capacity planning and consumer design.

By the end of this article, you'll be able to design a Kinesis pipeline that handles 10x your expected load without dropping records, diagnose shard hot-spotting in minutes, and avoid the cost traps that catch 90% of teams.

Shard Calculus: How Many Shards Do You Actually Need?

Every Kinesis shard provides 1 MB/s write and 2 MB/s read throughput. That's it. No bursting, no credits. If you exceed 1 MB/s on a shard for more than a few seconds, you get WriteProvisionedThroughputExceeded. The classic mistake is sizing shards based on average throughput, not peak. If your average is 5 MB/s but your peak is 20 MB/s for 10 seconds, you need 20 shards, not 5. Use the formula: shards = ceil(peak_write_throughput_MBps / 1.0). But also consider partition key distribution. A single hot partition key can max out one shard while others sit idle. Always use a high-cardinality partition key (like a UUID or a hash of the record). Never use timestamp alone—that creates a hot shard every second.

# io.thecodeforge — DevOps tutorial

import math

def calculate_shards(peak_mbps: float, avg_record_size_kb: float, records_per_second: int) -> int:
    """
    Calculate minimum shards for Kinesis Data Streams.
    Shard write limit: 1 MB/s or 1000 records/s, whichever is hit first.
    """
    throughput_shards = math.ceil(peak_mbps / 1.0)
    # 1000 records/s per shard
    record_rate_shards = math.ceil(records_per_second / 1000)
    return max(throughput_shards, record_rate_shards)

# Example: 10 MB/s peak, 2 KB avg record, 5000 records/s
shards = calculate_shards(10.0, 2, 5000)
print(f"Minimum shards: {shards}")  # Output: Minimum shards: 10

Producer Patterns: Reliable Ingestion Without Data Loss

The Kinesis Producer Library (KPL) is the recommended way to send data. It aggregates records into larger payloads (up to 1 MB) and uses a background thread to send them. But KPL has a gotcha: it buffers records in memory and on disk. If your process crashes before the buffer is flushed, you lose data. For critical pipelines, use the AWS SDK directly with synchronous putRecord calls and a retry queue. The KPL's RecordMaxBufferedTime default is 100 ms—too low for high throughput. Set it to 1000 ms to reduce API calls. Also, always enable AggregationEnabled (default true) to pack multiple records into one PutRecords request. For Firehose, the producer is simpler: just PutRecord to the delivery stream. Firehose automatically buffers and batches to S3/Redshift. But watch out: Firehose has a 60-second buffer interval by default. If you need lower latency, reduce it to 60 seconds minimum (cannot go lower).

# io.thecodeforge — DevOps tutorial

import boto3
import json
from botocore.config import Config

# Production config: retries with backoff, timeouts
config = Config(
    retries={'max_attempts': 5, 'mode': 'adaptive'},
    read_timeout=10,
    connect_timeout=5
)
kinesis = boto3.client('kinesis', config=config)

def send_event(stream_name: str, partition_key: str, event: dict) -> bool:
    """Send a single event with retry. Returns True on success."""
    try:
        response = kinesis.put_record(
            StreamName=stream_name,
            Data=json.dumps(event).encode('utf-8'),
            PartitionKey=partition_key
        )
        # Check for throttling
        if response['ResponseMetadata']['HTTPStatusCode'] == 200:
            return True
        else:
            print(f"Non-200 response: {response}")
            return False
    except Exception as e:
        print(f"PutRecord failed: {e}")
        return False

# Usage
send_event('my-stream', 'user-123', {'action': 'click', 'timestamp': 1234567890})

Consumer Design: Lambda vs. KCL vs. Firehose

You have three main consumer options: Lambda, Kinesis Client Library (KCL), and Firehose. Lambda is the simplest: it polls each shard and invokes your function with a batch of records. But Lambda has a concurrency limit (default 1000 per account) and each shard can only be processed by one Lambda invocation at a time. If your processing takes longer than the iterator age, you fall behind. The fix: increase Lambda batch size and concurrency, but also ensure your function is idempotent because Lambda may retry on failure. KCL (Java/KCL multi-lang daemon) gives you more control: it manages checkpointing, shard discovery, and load balancing across workers. Use KCL when you need exactly-once semantics or custom checkpointing. Firehose is the simplest: it reads from a stream and writes to S3/Redshift/Elasticsearch. No code needed. But you lose the ability to transform data in-flight (unless you use Lambda transformation, which adds latency).

# io.thecodeforge — DevOps tutorial

import base64
import json

def lambda_handler(event, context):
    """Process Kinesis records from a Lambda trigger."""
    for record in event['Records']:
        # Decode base64 data
        payload = base64.b64decode(record['kinesis']['data']).decode('utf-8')
        data = json.loads(payload)
        
        # Process the record (idempotent)
        process_event(data)
        
        # If processing fails, raise exception to trigger retry
        # Lambda will retry the entire batch unless you catch and skip
    return {'batchItemFailures': []}  # Return failed sequence numbers for partial success

def process_event(data):
    # Your business logic here
    pass

Error Handling and Retries: The 3 AM Wake-Up Call

Kinesis retries are brutal by default. If a Lambda consumer throws an error, Lambda retries the entire batch until the data expires (up to 7 days). This can cause a backlog that takes hours to clear. The fix: implement dead-letter queues (DLQ) for poison pills. Use a Lambda destination on failure to send the failed record to an SQS queue. Then process the DLQ separately. For KCL, handle ShutdownException and ThrottlingException by checkpointing only after successful processing. Never checkpoint before processing—if your worker crashes, you lose records. For Firehose, failed deliveries are retried for up to 24 hours, then sent to a backup S3 bucket. Always configure a backup bucket.

# io.thecodeforge — DevOps tutorial

import json
import boto3

sqs = boto3.client('sqs')
DLQ_URL = 'https://sqs.us-east-1.amazonaws.com/123456789012/kinesis-dlq'

def send_to_dlq(failed_record: dict, error: str):
    """Send a failed Kinesis record to SQS DLQ."""
    message = {
        'record': failed_record,
        'error': error,
        'timestamp': int(time.time())
    }
    sqs.send_message(
        QueueUrl=DLQ_URL,
        MessageBody=json.dumps(message)
    )

def lambda_handler(event, context):
    failed_sequences = []
    for record in event['Records']:
        try:
            process_record(record)
        except Exception as e:
            # Send to DLQ and skip (don't retry)
            send_to_dlq(record, str(e))
            failed_sequences.append(record['kinesis']['sequenceNumber'])
    return {'batchItemFailures': [{'itemIdentifier': seq} for seq in failed_sequences]}

Monitoring and Alerting: The Metrics That Matter

CloudWatch emits several Kinesis metrics. The critical ones: WriteProvisionedThroughputExceeded (shard throttling), ReadProvisionedThroughputExceeded (consumer throttling), MillisBehindLatest (consumer lag), and UserRecords.Success (ingestion rate). Set CloudWatch alarms on MillisBehindLatest > 5000 ms for 5 minutes. Also monitor GetRecords.IteratorAgeMilliseconds for Lambda consumers. For Firehose, monitor DeliveryToS3.Success and BackupToS3.Success. A common mistake is not setting up dashboards. Create a single pane of glass with shard count, throughput, iterator age, and error rates. Use Contributor Insights to find hot partition keys.

// io.thecodeforge — DevOps tutorial

{
  "AlarmName": "Kinesis-IteratorAge-High",
  "MetricName": "MillisBehindLatest",
  "Namespace": "AWS/Kinesis",
  "Statistic": "Maximum",
  "Period": 300,
  "EvaluationPeriods": 2,
  "Threshold": 5000,
  "ComparisonOperator": "GreaterThanThreshold",
  "AlarmActions": ["arn:aws:sns:us-east-1:123456789012:alert-topic"]
}

Cost Optimization: Don't Let Shards Drain Your Wallet

Kinesis Data Streams charges per shard-hour ($0.015 per shard-hour in us-east-1). A 100-shard stream costs $1.50/hour = $1,080/month. That's before data transfer and API costs. The biggest cost trap: over-provisioning shards for peak throughput and leaving them running 24/7. Use auto-scaling: monitor WriteProvisionedThroughputExceeded and scale up during peaks, scale down during lulls. AWS provides a shard-level CloudWatch metric IncomingBytes and IncomingRecords. Use Application Auto Scaling with a target tracking policy on IncomingBytes (target 0.8 MB/s per shard). For Firehose, costs are based on data volume ($0.029/GB) and S3 storage. Firehose is cheaper than Kinesis Streams for simple delivery. For Data Analytics, costs are per Kinesis Processing Unit (KPU) at $0.11/hour. Optimize Flink jobs by tuning parallelism and state backend.

// io.thecodeforge — DevOps tutorial

{
  "PolicyName": "kinesis-shard-scaling",
  "ServiceNamespace": "kinesis",
  "ResourceId": "stream/my-stream",
  "ScalableDimension": "kinesis:stream:WriteCapacityUnits",
  "TargetTrackingScalingPolicyConfiguration": {
    "TargetValue": 0.8,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "KinesisWriteProvisionedThroughputExceeded"
    },
    "ScaleOutCooldown": 60,
    "ScaleInCooldown": 300
  }
}

When Not to Use Kinesis: The Alternatives

Kinesis is not a message queue. If you need individual message delivery, exactly-once semantics, or fan-out to many consumers, use SQS or SNS. Kinesis is a stream: all consumers read the same data in order. If you need to broadcast to multiple independent consumers, each with their own offset, Kinesis works (each consumer group gets its own shard iterator), but Kafka is cheaper for high-throughput fan-out. For very low latency (< 100 ms), consider AWS IoT Core or WebSocket APIs. For batch processing, use S3 events with Lambda. Kinesis is overkill if your throughput is < 1 MB/s and you don't need replay. A simple Lambda with SQS might be cheaper and simpler.