Cloud Functions: 2nd Gen, Eventarc, and Background Functions

You deployed a Cloud Function to process Pub/Sub messages. It worked in dev. In prod, it silently dropped 30% of events because the 1st gen 9-minute timeout hit during a spike. No retries. No DLQ. Just lost data. That's the reality of 1st gen Cloud Functions: a toy for prototypes, not production. Cloud Functions 2nd Gen fixes this by running on Cloud Run under the hood, giving you 60-minute timeouts, up to 16 vCPUs, 16 GiB memory, and native Eventarc integration. But with power comes complexity: you now manage concurrency, cold starts, and IAM across multiple services. This guide walks you through building a production-ready event-driven pipeline using 2nd Gen, Eventarc, and background functions — no fluff, just patterns that survive Black Friday.

Why 2nd Gen Exists: The 1st Gen Pain Points

First-gen Cloud Functions were designed for simple, low-volume tasks. They max out at 9 minutes timeout, 2 GiB memory, and 1 vCPU. More critically, they lack native integration with Eventarc — you had to manually wire Pub/Sub topics to functions, and retries were basic. In production, we saw functions timeout during batch processing, events lost due to lack of dead-letter queues, and cold starts exceeding 10 seconds under load. 2nd Gen solves this by wrapping Cloud Run: you get the same serverless experience but with Cloud Run's scalability and Eventarc's event management. The trade-off? You now pay for the underlying Cloud Run container instance when idle (though the free tier covers small workloads). For any serious event-driven architecture, 2nd Gen is the minimum viable option.

gcloud functions deploy my-function \
  --runtime nodejs20 \
  --trigger-topic my-topic \
  --timeout 540 \
  --memory 2048MB \
  --max-instances 10

Eventarc: The Event Router You Didn't Know You Needed

Eventarc is Google Cloud's eventing backbone. It ingests events from over 90 sources (Cloud Storage, Pub/Sub, Audit Logs, etc.) and routes them to destinations like Cloud Run, Cloud Functions, or Workflows. With 2nd Gen, your function becomes an Eventarc trigger — you define the event source and filtering criteria in a single YAML or gcloud command. Eventarc handles retries with exponential backoff, dead-letter queues, and ordering keys. This replaces the manual Pub/Sub wiring of 1st Gen. For example, you can trigger a function only on 'storage.object.finalize' events from a specific bucket with a prefix filter. Eventarc also supports custom events via channels, enabling event-driven architectures across services.

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: my-trigger
spec:
  broker: default
  filter:
    attributes:
      type: google.cloud.storage.object.v1.finalized
      bucket: my-prod-bucket
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: my-function
    uri: /process-file

Background Functions: The Event-Driven Workhorse

Background functions are Cloud Functions triggered by Eventarc events (not HTTP). They receive a CloudEvent payload and process it asynchronously. In 2nd Gen, background functions are essentially Cloud Run services with an Eventarc trigger. You write a function that accepts a CloudEvent object, processes it, and returns nothing (or an error for retry). The key difference from HTTP functions: you don't control the response — Eventarc handles acknowledgment. If your function throws an exception, Eventarc retries based on the retry policy (default: 5 retries over 24 hours). You can configure dead-letter topics for failed events. Background functions are ideal for data processing pipelines, audit log analysis, and any task that doesn't need a synchronous response.

import functions_framework
from cloudevents.http import CloudEvent

@functions_framework.cloud_event
def process_file(event: CloudEvent):
    data = event.data
    bucket = data['bucket']
    name = data['name']
    print(f"Processing file: gs://{bucket}/{name}")
    # Simulate processing
    if name.endswith('.csv'):
        raise ValueError("CSV files not supported")

Deploying a 2nd Gen Background Function with Eventarc

Deploying a 2nd Gen function is straightforward with gcloud. You specify --gen2, the event trigger via Eventarc, and the function source. The deployment creates a Cloud Run service and an Eventarc trigger. Key flags: --trigger-event-filters (e.g., type, bucket), --trigger-location (must match bucket region), --retry (enable retries), --max-instances (control concurrency). Unlike 1st Gen, you can set --concurrency (default 80) and --cpu (up to 8). The function must be idempotent because Eventarc may deliver events at least once. Use event IDs to deduplicate. Also, set --timeout up to 3600 seconds. After deployment, monitor with Cloud Logging and Error Reporting.

gcloud functions deploy process-file \
  --gen2 \
  --runtime python312 \
  --region us-central1 \
  --source . \
  --entry-point process_file \
  --trigger-event-filters type=google.cloud.storage.object.v1.finalized \
  --trigger-event-filters bucket=my-prod-bucket \
  --trigger-location us \
  --retry \
  --max-instances 10 \
  --concurrency 20 \
  --cpu 2 \
  --memory 4Gi \
  --timeout 600

Event Filtering: Don't Process What You Don't Need

Eventarc supports attribute-based filtering on event type, subject, and custom attributes. This is critical for cost and performance. For example, if you only care about CSV files in a specific bucket, filter on bucket and object suffix. Filters are evaluated before the function is invoked, so no wasted invocations. You can also use Cloud Audit Logs as an event source — filter on service name, method, and resource type. This is powerful for compliance and security automation. However, filters are limited to exact matches and prefix/suffix patterns. For complex logic, use a lightweight router function that inspects the event and forwards to specific handlers.

gcloud functions deploy process-csv \
  --gen2 \
  --runtime nodejs20 \
  --trigger-event-filters type=google.cloud.storage.object.v1.finalized \
  --trigger-event-filters bucket=my-prod-bucket \
  --trigger-event-filters-path-pattern object=csv-files/*.csv \
  --trigger-location us

Retries and Dead-Letter Queues: Handling Failures Gracefully

Eventarc retries failed deliveries with exponential backoff (up to 5 retries over 24 hours by default). You can configure the retry policy via the Eventarc trigger or the function's --retry flag. But retries alone aren't enough — you need a dead-letter queue (DLQ) for events that fail permanently. To set up a DLQ, create a Pub/Sub topic and configure the Eventarc trigger to send failed events there. Then, have a separate function process the DLQ (e.g., log, alert, or reprocess after manual fix). This pattern prevents event loss and provides observability. In production, we monitor DLQ depth and set alerts for spikes.

gcloud pubsub topics create failed-events

gcloud functions deploy process-file \
  --gen2 \
  --trigger-event-filters type=google.cloud.storage.object.v1.finalized \
  --trigger-event-filters bucket=my-prod-bucket \
  --retry \
  --dead-letter-topic failed-events \
  --max-retry-attempts 3 \
  --min-retry-delay 10s \
  --max-retry-delay 600s

Cold Starts and Concurrency: Taming the Beast

2nd Gen functions run on Cloud Run, which means cold starts still exist but are manageable. Cloud Run keeps instances warm if they receive traffic regularly. For background functions, concurrency is key: you can set --concurrency to control how many events a single instance processes simultaneously. Higher concurrency reduces cold starts but increases memory pressure. For CPU-bound tasks, set concurrency low (1-5). For I/O-bound tasks, set it high (up to 80). Also, use min-instances to keep a baseline of warm instances, but this costs money. In production, we use min-instances=1 for latency-sensitive functions and rely on concurrency for throughput.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: process-file
spec:
  template:
    spec:
      containerConcurrency: 20
      containers:
      - image: gcr.io/my-project/process-file
        resources:
          limits:
            cpu: 2
            memory: 4Gi
      minInstanceCount: 1

IAM and Security: Least Privilege for Eventarc

Eventarc triggers require specific IAM permissions. The Eventarc service account needs permission to invoke your function. The function's runtime service account needs access to event sources (e.g., Cloud Storage, Pub/Sub). Use custom service accounts with minimal roles. For example, a function that reads from a bucket needs storage.objectViewer, not storage.admin. Also, use VPC Service Controls to restrict data exfiltration. In production, we create a dedicated service account per function and grant only the required roles. Audit logs should be enabled to track invocations.

gcloud iam service-accounts create process-file-sa \
  --display-name "Process File Function SA"

gcloud projects add-iam-policy-binding my-project \
  --member serviceAccount:process-file-sa@my-project.iam.gserviceaccount.com \
  --role roles/storage.objectViewer

gcloud functions deploy process-file \
  --gen2 \
  --service-account process-file-sa@my-project.iam.gserviceaccount.com

Monitoring and Observability: Know When Things Break

2nd Gen functions emit metrics to Cloud Monitoring: invocation count, latency, error count, and instance count. Set up dashboards and alerts for error rate > 1%, latency p99 > 10s, and DLQ depth > 0. Use Cloud Logging to view function logs with structured logging. For background functions, log the event ID and processing status. Also, use Error Reporting to group and notify on exceptions. In production, we have a Slack alert when DLQ depth exceeds 10. Without monitoring, you'll discover failures from customer complaints.

import logging
import google.cloud.logging

client = google.cloud.logging.Client()
client.setup_logging()

def process_file(event):
    event_id = event['id']
    logging.info(f"Processing event {event_id}", extra={"event_id": event_id})
    try:
        # processing logic
        pass
    except Exception as e:
        logging.error(f"Failed to process event {event_id}: {e}", extra={"event_id": event_id})
        raise

Cost Optimization: Pay Only for What You Use

2nd Gen functions are priced based on Cloud Run: you pay for vCPU-seconds, memory-seconds, and requests. Background functions also incur Eventarc costs per event. To optimize: use concurrency to maximize throughput per instance, set appropriate memory (don't overprovision), and use min-instances sparingly. Also, filter events aggressively to reduce invocations. For high-volume pipelines, consider using Cloud Run jobs instead of functions for batch processing. In production, we reduced costs by 40% by right-sizing memory and concurrency based on load testing.

# Estimate monthly cost for 10M invocations, 2 vCPU, 4GiB, 1s avg duration
# Cloud Run: ~$0.000024 per vCPU-second, ~$0.0000025 per GiB-second
# 10M * 1s * (2*0.000024 + 4*0.0000025) = $0.58
# Eventarc: $0.00 (first 2M events free), then $0.40 per million
# Total: ~$3.78/month

Migrating from 1st Gen to 2nd Gen: A Practical Guide

Migration involves redeploying your function with --gen2 and updating the trigger to Eventarc. First, review your function's timeout and memory requirements — 2nd Gen supports higher limits. Update your code to use the CloudEvent format (functions_framework.cloud_event). Test in a staging environment with mirrored traffic. Use Eventarc's ordering keys if you need event ordering (not guaranteed by default). After migration, monitor for regressions. Rollback by redeploying the 1st Gen version. In production, we migrated 50 functions over a weekend with zero downtime by using traffic splitting on Cloud Run.

# Before (1st Gen)
def process_file(data, context):
    bucket = data['bucket']
    name = data['name']
    
# After (2nd Gen)
import functions_framework
from cloudevents.http import CloudEvent

@functions_framework.cloud_event
def process_file(event: CloudEvent):
    data = event.data
    bucket = data['bucket']
    name = data['name']

Real-World Architecture: Event-Driven Data Pipeline

Let's tie it all together with a real architecture: a data pipeline that processes CSV files uploaded to Cloud Storage. When a CSV lands, Eventarc triggers a 2nd Gen background function that validates the file, transforms it, and writes to BigQuery. If validation fails, the function publishes to a DLQ. A separate function monitors the DLQ and alerts Slack. The pipeline uses concurrency=10, min-instances=1, and a custom SA with storage.objectViewer and bigquery.dataEditor. This architecture handles millions of files per day with 99.99% reliability.

import functions_framework
from cloudevents.http import CloudEvent
from google.cloud import bigquery, storage

@functions_framework.cloud_event
def process_csv(event: CloudEvent):
    data = event.data
    bucket = data['bucket']
    name = data['name']
    if not name.endswith('.csv'):
        return  # filter non-CSV
    # Download and validate
    client = storage.Client()
    blob = client.bucket(bucket).blob(name)
    content = blob.download_as_text()
    if not content.startswith('header'):
        raise ValueError("Invalid CSV header")
    # Transform and load to BigQuery
    bq = bigquery.Client()
    table_id = "my-project.dataset.files"
    rows = [{"filename": name, "content": content}]
    errors = bq.insert_rows_json(table_id, rows)
    if errors:
        raise RuntimeError(f"BigQuery insert failed: {errors}")

Pub/Sub Acknowledgement Deadline: Preventing Duplicate Executions

Event-driven 2nd Gen functions use Eventarc with an underlying Pub/Sub subscription. By default, the Pub/Sub acknowledgement (ack) deadline is 10 seconds. If your function takes longer than 10 seconds to process an event, Pub/Sub assumes the function failed and redelivers the message, causing duplicate executions. For functions with processing times exceeding 10 seconds—common with file processing, API calls, or ML inference—you must increase the ack deadline. Set it to the maximum of 600 seconds via the gcloud eventarc triggers update command. The ack deadline should match or exceed your function's maximum expected execution time plus a buffer. Failure to configure this is one of the top causes of 'duplicate event' bugs in production.

# Get the Pub/Sub subscription name for an Eventarc trigger
TRIGGER_NAME="my-trigger"
REGION="us-central1"
SUBSCRIPTION=$(gcloud eventarc triggers describe $TRIGGER_NAME \
  --location=$REGION \
  --format="value(transport.pubsub.subscription)")

# Update the ack deadline to 600 seconds (max)
gcloud pubsub subscriptions update $SUBSCRIPTION \
  --ack-deadline=600

Writing Event-Driven Functions: CloudEvents Format and Function Termination

2nd Gen event-driven functions use the CloudEvents specification. Your function must accept a CloudEvent object (not the old background function signature with 'data' and 'context'). The Functions Framework for each runtime handles unmarshalling CloudEvents. Key rules: your function must terminate all background tasks (threads, promises, callbacks, system processes) before returning. Any unresolved tasks may not complete and cause undefined behavior. Use async/await or promise chaining to ensure completion. For HTTP functions, you send an HTTP response; for event-driven functions, the return signals acknowledgement. If you throw an exception, Eventarc retries based on the retry policy. Always catch expected exceptions and log them with structured logging using the event ID for correlation across retries.

import functions_framework
from cloudevents.http import CloudEvent

@functions_framework.cloud_event
def process_file(event: CloudEvent) -> None:
    event_id = event['id']
    data = event.data
    bucket = data['bucket']
    name = data['name']
    try:
        # Processing logic here
        result = process_data(bucket, name)
        print(f"Successfully processed {name} (event {event_id})")
    except ValueError as e:
        # Expected error: log and don't retry
        print(f"Invalid data for {name}: {e}")
        # Don't re-raise - this prevents Eventarc retries
    except Exception as e:
        # Unexpected error: re-raise to trigger retry
        print(f"Unexpected error processing {name}: {e}")
        raise