AWS Lambda Cold Starts — Why P99 Spikes to 1.2s at 9 AM

Every application needs compute power — something has to run your code. Traditionally, that meant renting a virtual machine or physical server that runs 24/7, even at 3 a.m. when zero users are online. You're paying for potential, not actual work. As cloud adoption exploded, this idle-cost problem became impossible to ignore, especially for startups and teams with unpredictable traffic spikes.

AWS Lambda, launched in 2014, flipped the model. Instead of managing servers, you upload a function — a single, focused piece of logic — and AWS handles everything else: provisioning, scaling, patching, and availability. The term 'serverless' doesn't mean there are no servers; it means YOU don't manage them. The servers exist, they're just Amazon's problem. This lets your team focus entirely on business logic instead of infrastructure operations.

By the end of this article you'll understand how Lambda executes code, how to wire it to real-world triggers like API Gateway and S3, how to avoid the cold start trap that kills performance, and how to structure a production-worthy serverless workflow. You'll also know exactly when Lambda is the right tool — and when it absolutely isn't.

How AWS Lambda Actually Executes Your Code — The Execution Model

Lambda's execution model is the foundation everything else builds on. When a trigger fires — say, an HTTP request hits API Gateway — Lambda needs to run your function. If a pre-warmed container exists from a recent invocation, Lambda reuses it. This is a 'warm start' and it's fast. If no container is available, Lambda has to bootstrap one from scratch: download your code package, spin up a runtime environment, run any initialisation code outside your handler, then finally invoke your handler. That bootstrap phase is the dreaded cold start.

Cold starts typically add 100ms–1000ms of latency depending on the runtime (.NET and Java are heavier; Node.js and Python are lighter). For a background job this is irrelevant. For a user-facing API call, it's noticeable.

Your handler function receives two objects: the event (the payload that triggered the invocation — could be an HTTP body, an S3 event, a queue message) and the context (metadata about the invocation itself — function name, memory limit, request ID). Understanding this distinction is critical: the event is about WHAT happened, the context is about WHO is running.

Code outside the handler runs once per container lifecycle. That's where you put database connections, SDK clients, and config loading — doing it inside the handler means re-initialising on every single invocation, which is both slow and wasteful.

import boto3
import json
import os
from PIL import Image
import io

# ✅ Initialise the S3 client OUTSIDE the handler.
# This runs once when the container boots (cold start),
# then gets reused across all warm invocations — saving ~50ms per call.
s3_client = boto3.client('s3')

# Target width for all resized thumbnails
THUMBNAIL_WIDTH = 200


def handler(event, context):
    """
    Triggered by an S3 PUT event whenever a new image is uploaded
    to the 'uploads-raw' bucket. Resizes it and saves a thumbnail
    to the 'uploads-thumbnails' bucket.
    """

    # The event payload from S3 contains a list of records —
    # each record represents one file upload.
    for record in event['Records']:
        source_bucket = record['s3']['bucket']['name']
        object_key    = record['s3']['object']['key']  # e.g. 'photos/sunset.jpg'

        print(f"Processing: s3://{source_bucket}/{object_key}")

        # Download the original image bytes into memory (no temp file needed)
        response      = s3_client.get_object(Bucket=source_bucket, Key=object_key)
        image_bytes   = response['Body'].read()

        # Open image with Pillow and calculate proportional height
        original_img  = Image.open(io.BytesIO(image_bytes))
        original_w, original_h = original_img.size
        ratio         = THUMBNAIL_WIDTH / original_w
        new_height    = int(original_h * ratio)

        thumbnail     = original_img.resize((THUMBNAIL_WIDTH, new_height))

        # Save resized image to an in-memory buffer — Lambda has no persistent disk
        output_buffer = io.BytesIO()
        thumbnail.save(output_buffer, format='JPEG', quality=85)
        output_buffer.seek(0)  # Rewind buffer to the start before uploading

        # Write thumbnail to the destination bucket under the same key name
        destination_bucket = os.environ['THUMBNAIL_BUCKET']  # Read from env vars, not hardcoded
        s3_client.put_object(
            Bucket      = destination_bucket,
            Key         = object_key,
            Body        = output_buffer,
            ContentType = 'image/jpeg'
        )

        print(f"Thumbnail saved: s3://{destination_bucket}/{object_key} ({THUMBNAIL_WIDTH}x{new_height})")

    # Lambda expects a return value when invoked synchronously (e.g. via API Gateway).
    # For async triggers like S3, the return value is ignored — but it's good practice.
    return {
        'statusCode': 200,
        'body': json.dumps({'processed': len(event['Records'])})
    }

Wiring Lambda to the Real World — Triggers, Events, and API Gateway

A Lambda function sitting alone does nothing. It needs a trigger — an AWS service that says 'hey, something happened, go run'. The trigger determines the shape of the event object your handler receives, which is why reading the AWS event schema docs for each trigger type matters.

The most common triggers in production are: API Gateway (HTTP requests), S3 (file uploads/deletions), SQS (queue messages for async processing), EventBridge (scheduled cron jobs and event routing), DynamoDB Streams (react to database changes), and SNS (fan-out notifications).

API Gateway is the one you'll use for building REST APIs or webhooks. When a request hits your endpoint, API Gateway wraps it into a structured event object and hands it to Lambda. Your function returns a response object with a statusCode, headers, and body, and API Gateway translates that back into a real HTTP response.

The Lambda Proxy Integration model (the default and recommended approach) passes the raw request to your function and expects you to construct the full HTTP response yourself. This gives you complete control over status codes, CORS headers, and response bodies. Older tutorials show Lambda custom integrations — avoid them, they're fiddly and add complexity for no gain.

For async workloads, SQS is your best friend. Rather than calling Lambda directly (which creates tight coupling), push messages to a queue and let Lambda poll and process them in batches. This naturally handles traffic bursts without rate-limit errors.

import json
import boto3
import uuid
import os
from datetime import datetime, timezone

# DynamoDB resource initialised at cold-start — reused on warm invocations
dynamodb    = boto3.resource('dynamodb')
orders_table = dynamodb.Table(os.environ['ORDERS_TABLE_NAME'])


def handler(event, context):
    """
    Handles POST /orders from API Gateway (Lambda Proxy Integration).
    Creates a new order record in DynamoDB and returns the order ID.

    API Gateway event shape (key fields):
      event['httpMethod']         -> 'POST'
      event['path']               -> '/orders'
      event['body']               -> Raw JSON string of the request body
      event['requestContext']     -> Metadata including caller identity
    """

    http_method = event.get('httpMethod', '')

    # Route guard — this function only handles order creation
    if http_method != 'POST':
        return _build_response(405, {'error': f'Method {http_method} not allowed'})

    # API Gateway sends the body as a raw string — we must parse it
    try:
        request_body = json.loads(event.get('body') or '{}')
    except json.JSONDecodeError:
        return _build_response(400, {'error': 'Request body must be valid JSON'})

    # Validate required fields before touching the database
    required_fields = ['customer_id', 'items', 'total_amount']
    missing_fields  = [f for f in required_fields if f not in request_body]
    if missing_fields:
        return _build_response(400, {'error': f'Missing required fields: {missing_fields}'})

    # Build the order record
    order_id    = str(uuid.uuid4())  # Unique ID for this order
    created_at  = datetime.now(timezone.utc).isoformat()  # ISO 8601, always UTC

    order_record = {
        'order_id':      order_id,
        'customer_id':   request_body['customer_id'],
        'items':         request_body['items'],
        'total_amount':  str(request_body['total_amount']),  # DynamoDB doesn't support float natively
        'status':        'PENDING',
        'created_at':    created_at
    }

    # Write to DynamoDB — put_item overwrites if the key already exists,
    # so ConditionExpression ensures we never silently stomp an existing order
    orders_table.put_item(
        Item=order_record,
        ConditionExpression='attribute_not_exists(order_id)'
    )

    print(f"Order created: {order_id} for customer {request_body['customer_id']}")

    return _build_response(201, {
        'order_id':   order_id,
        'status':     'PENDING',
        'created_at': created_at
    })


def _build_response(status_code, body_dict):
    """
    Constructs the response object API Gateway expects.
    CORS headers are included so browser-based clients can call this API.
    Without these headers, browsers silently block the response.
    """
    return {
        'statusCode': status_code,
        'headers': {
            'Content-Type':                'application/json',
            'Access-Control-Allow-Origin': '*'  # Tighten to your domain in production
        },
        'body': json.dumps(body_dict)
    }

Lambda Event Source Reference Table — What Triggers Your Function

The following table catalogs the most common Lambda event sources, their invocation model, payload size limits, retry behavior, and best-fit use cases. Knowing these details helps you design reliable, cost-efficient serverless workflows. For each source, the event structure is fixed by AWS — you cannot change the schema — so you must parse the documented fields correctly in your handler.

Key details to remember: - Asynchronous invocations (S3, SNS, EventBridge) retry twice with 1–2 minute delays. Always configure a dead-letter queue (DLQ) for these triggers. - Stream-based triggers (DynamoDB, Kinesis) retry until the data record expires — a persistent bug will block the entire shard. Use bisectBatchOnFunctionError to split batches on failure. - Synchronous triggers (API Gateway, Lambda Function URL) do not retry; your client or upstream service must implement retry logic. - Payload size limits are hard: if your S3 event payload exceeds 128 KB, S3 will send the notification anyway but truncates the event — use the Deep Archive storage class sparingly to avoid this.

For a full list of event sources and their exact event schemas, refer to the [AWS Lambda Developer Guide — Using AWS Lambda with other services](https://docs.aws.amazon.com/lambda/latest/dg/lambda-services.html).

Cold Starts, Memory Tuning, and the Performance Levers You Actually Control

Lambda gives you one direct performance dial: memory. You set it anywhere from 128 MB to 10,240 MB. What most developers don't realise is that CPU allocation scales proportionally with memory. A 1,024 MB Lambda function gets roughly 8x the CPU of a 128 MB one. If your function is CPU-bound (image processing, data transformation, encryption), doubling the memory can halve the execution time — and since you pay for duration × memory, the cost often stays the same or even drops.

Cold starts are the other major lever. Three strategies exist: Provisioned Concurrency, keeping functions warm with scheduled EventBridge pings, and minimising package size.

Provisioned Concurrency is the only AWS-supported solution. You pay for a set number of pre-warmed containers to stay alive at all times. It costs more than on-demand but eliminates cold starts entirely for that concurrency slot. Use it for customer-facing APIs where tail latency matters.

Package size matters because Lambda has to download your deployment package before running it. A 50 MB Python package with unnecessary dependencies cold-starts noticeably slower than a 3 MB lean package. Use Lambda Layers to separate large dependencies (like numpy or Pillow) from your application code, and use .zip deployment packages rather than container images unless you specifically need Docker tooling.

Finally, watch your timeout setting. The default is 3 seconds. Downstream API calls, DB queries, and S3 operations can easily exceed this. Set it realistically (15 minutes max) and always handle partial failures gracefully.

# AWS SAM (Serverless Application Model) template — the standard way to
# define Lambda functions as Infrastructure-as-Code.
# Run: sam build && sam deploy --guided

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Description: Orders API — production-grade Lambda configuration

Globals:
  Function:
    # Runtime for all functions in this template unless overridden
    Runtime: python3.12
    # Timeout generous enough for DynamoDB + downstream calls, not infinite
    Timeout: 30
    # Environment variables available to all functions
    Environment:
      Variables:
        LOG_LEVEL:          INFO
        ORDERS_TABLE_NAME:  !Ref OrdersTable

Resources:

  OrdersApiFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: orders-api-handler
      CodeUri:       src/
      Handler:       orders_api_handler.handler

      # 512 MB gives ~4x CPU vs 128 MB — worth it for JSON parsing + DynamoDB calls
      # Run AWS Lambda Power Tuning tool to find YOUR optimal memory setting
      MemorySize: 512

      # Provisioned concurrency: 5 containers always warm for the production alias
      # This eliminates cold starts for the first 5 concurrent requests
      # Cost: ~$0.0000041 per GB-second × 5 containers × all hours in month
      AutoPublishAlias: live
      ProvisionedConcurrencyConfig:
        ProvisionedConcurrentExecutions: 5

      # IAM permissions — principle of least privilege
      # Only grant what this specific function actually needs
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref OrdersTable

      # API Gateway trigger — Lambda Proxy Integration (recommended)
      Events:
        CreateOrder:
          Type:  Api
          Properties:
            Path:   /orders
            Method: POST

        # OPTIONS method needed for browser CORS preflight requests
        CreateOrderOptions:
          Type:  Api
          Properties:
            Path:   /orders
            Method: OPTIONS

  OrdersTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName:    orders
      BillingMode:  PAY_PER_REQUEST  # Serverless billing — no provisioned capacity to manage
      AttributeDefinitions:
        - AttributeName: order_id
          AttributeType: S
      KeySchema:
        - AttributeName: order_id
          KeyType:        HASH

      # Point-in-time recovery — always enable for production data
      PointInTimeRecoverySpecification:
        PointInTimeRecoveryEnabled: true

Outputs:
  OrdersApiEndpoint:
    Description: "API Gateway endpoint for the Orders API"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/orders"

Provisioned Concurrency vs Cold Start — Visual Breakdown

Provisioned Concurrency is the only AWS-native mechanism that guarantees zero cold starts for a fixed number of concurrent invocations. The diagram below contrasts the request flow for an on-demand function (which may incur a cold start) versus a function with Provisioned Concurrency.

How it works: When you enable Provisioned Concurrency, Lambda pre-initialises a specified number of execution environments and keeps them warm. Incoming invocations are routed to these warm environments instantly. On-demand environments are still used for invocations beyond the provisioned count, so cold starts still occur when the provisioned pool is exhausted. The visual logic flow:

• On-Demand Path: Request arrives → check for warm container → if none found → cold start (init + handler delay).
• Provisioned Concurrency Path: Request arrives → route to pre-warmed container → warm start (handler only, no init delay).

The benefit is a 100% elimination of cold start latency for the initial set of concurrent requests. The cost is paying for those environments even when idle.

When to use it: Only for latency-critical production endpoints where p99 must stay below, say, 500ms. For batch processing or background jobs, on-demand is sufficient and cheaper.

When NOT to use it: If your function is rarely invoked (once per hour), the cost of keeping a container warm 24/7 will far exceed any performance benefit. A simple scheduled EventBridge ping (every 5 minutes) is cheaper and nearly as effective — though not guaranteed, as AWS may reclaim containers during maintenance.

Alternative warming patterns: A common pattern is to set up an EventBridge rule that invokes your function every 5 minutes with a synthetic event (e.g., a 'warmup' field). This keeps 1–2 containers warm without Provisioned Concurrency cost. However, this is unreliable under burst traffic — if multiple concurrent requests arrive simultaneously, only one container may be warm. Provisioned Concurrency guarantees capacity.

# Snippet: enabling Provisioned Concurrency in SAM
# Full template in previous section; this is the critical part

      AutoPublishAlias: live
      ProvisionedConcurrencyConfig:
        ProvisionedConcurrentExecutions: 5

# After deploy, verify with:
# aws lambda get-provisioned-concurrency-config --function-name orders-api-handler --qualifier live
# Expected: Status: READY, AllocatedProvisionedConcurrentExecutions: 5

Lambda Resource Limits & Constraints Table — What You Can't Change

Lambda has specific hard limits that constrain how you design your serverless applications. Exceeding these limits results in deployment failures, throttling, or runtime errors. The table below shows the most important limits — know them before you architect your system.

How to work around limits: - Package size: If you exceed 250 MB unzipped, separate large libraries (Panda, OpenCV, etc.) into Lambda Layers. Each layer can be up to 250 MB, and you can use up to 5 layers, giving you an effective 1.25 GB total. - Timeout: Lambda supports up to 15 minutes. For longer jobs, use AWS Step Functions to orchestrate multiple Lambda calls, or switch to Fargate/Batch. - Concurrency: If you anticipate more than 1,000 concurrent executions, request a limit increase in the AWS Service Quotas console. Also consider using SQS buffering to smooth traffic. - Payload size: For payloads larger than 256 KB, upload to S3 and pass the object key in the event. Lambda reads from S3 instead of the event body.

These limits are not negotiable — building against them from day one avoids costly refactors later.

Production Patterns: Error Handling, Retries, and Observability

Lambda's default retry behaviour depends on invocation type. Synchronous invocations (API Gateway, custom apps) do NOT retry automatically — your client must handle errors. Asynchronous invocations (S3, SNS, EventBridge) retry twice using built-in retry logic, then discard the event unless you configure a dead-letter queue (DLQ). Stream-based triggers (DynamoDB Streams, Kinesis) retry until the data expires (default 24 hours) and block the shard — meaning a permanently failing function stalls your stream.

For synchronous APIs, implement your own retry with exponential backoff inside Lambda. For async triggers, always attach a DLQ (SQS or SNS) to capture failed events. Without a DLQ, failed events vanish after two retries — you'll never know.

Observability in Lambda is driven by CloudWatch Logs, CloudWatch Metrics, and AWS X-Ray. Every invocation writes a REPORT line showing duration, billed duration, memory used, and init duration. X-Ray traces show downstream calls to DynamoDB, S3, and other services — essential for debugging latency.

Structured logging is critical. Use JSON-formatted logs with a correlation ID (often the X-Ray trace ID) so you can correlate invocations. Avoid print() statements without context.

import json
import logging
import os
import traceback

# Configure structured JSON logging
logging.basicConfig(level=logging.INFO, format='%(message)s')
logger = logging.getLogger()


def handler(event, context):
    # Capture X-Ray trace ID for correlation
    trace_id = context.aws_request_id
    logger.info(json.dumps({
        'trace_id': trace_id,
        'event_type': type(event).__name__,
        'message': 'Function invoked'
    }))

    try:
        # business logic
        result = process_order(event)
        return _build_response(200, result)
    except ValidationError as e:
        logger.warning(json.dumps({
            'trace_id': trace_id,
            'error': str(e),
            'message': 'Validation failed'
        }))
        return _build_response(400, {'error': str(e)})
    except ExternalServiceError as e:
        logger.error(json.dumps({
            'trace_id': trace_id,
            'error': str(e),
            'message': 'Downstream service failed'
        }))
        # Retry with exponential backoff (simplified)
        time.sleep(2 ** context.retry_attempt)  # not recommended for sync; use async DLQ instead
        raise  # Let Lambda retry if async
    except Exception:
        logger.critical(json.dumps({
            'trace_id': trace_id,
            'error': traceback.format_exc(),
            'message': 'Unhandled exception'
        }))
        return _build_response(500, {'error': 'Internal server error'})

def _build_response(status_code, body):
    return {
        'statusCode': status_code,
        'headers': {'Content-Type': 'application/json'},
        'body': json.dumps(body)
    }

When Lambda is the Wrong Tool — Alternatives and Trade-offs

Lambda excels at short-lived, event-driven, bursty workloads. But it's not a general-purpose compute platform. If your workload contradicts any of the following, reach for another service.

First, long-running processes: Lambda's hard 15-minute timeout means you cannot run a nightly batch job that takes an hour. Use AWS Batch or ECS/Fargate for that.

Second, stateful applications: Lambda is stateless by design. If your application needs to hold client connections (WebSockets), maintain session state in memory, or use files that persist beyond a single invocation, you'll fight the architecture. Use EC2 or ECS with sticky sessions instead.

Third, predictable, steady traffic: If your load is constant 24/7, Lambda's per-ms billing is more expensive than a low-cost EC2 instance or a reserved instance. A t3.small running 24 hours costs $15/month; 5 million Lambda invocations at 200ms average could cost $8, but steady traffic at 100 req/s would push cost higher than an EC2.

Fourth, heavy GPU/compute: Lambda has no GPU support. ML training, 3D rendering, or video transcoding with high compute needs are better on EC2 GPU instances or SageMaker.

Fifth, very low latency requirements (<10ms): Lambda's cold start and network overhead make it unsuitable for sub-millisecond use cases like real-time trading. Use containers on EC2 or custom hardware.

Finally, large binary processing: Lambda's deployment package limit is 250 MB (unzipped) and 50 MB (zipped) for direct upload. If you're processing multi-GB files, you'll hit storage and timeout limits. Use ECS or Batch with EFS.

# Quick decision reference for choosing Lambda or an alternative

deployment_type:
  - name: Lambda
    good_for:
      - Event-driven functions
      - Spiky HTTP APIs
      - Scheduled tasks under 15 minutes
    bad_for:
      - Long-running batch jobs (over 15 min)
      - Stateful applications
      - Predictable steady traffic
  - name: ECS/Fargate
    good_for:
      - Containerized workloads (stateful or stateless)
      - Long-running services
      - WebSocket servers
      - Background workers needing >15 min
    bad_for:
      - Very short-lived functions (cold start is higher)
      - Simple event reactions (overkill)
  - name: EC2
    good_for:
      - Full control of OS/GPU
      - Stable traffic patterns
      - Applications needing persistent storage
    bad_for:
      - Variable traffic (idle cost)
      - Need minimal ops overhead

# Cost comparison example:
# Scenario: 100 req/s steady, average duration 100ms, 512 MB memory
# Lambda: 100 * 86400 = 8.64M invocations/day -> $1.728/day = $51.84/month
# t3.small (2 vCPU, 2 GB) on-demand: ~$15/month (24/7)
# Lambda is 3x more expensive for steady traffic.
# Lambda wins when traffic is spiky and idle periods exist.