AWS Glue: Serverless ETL and Data Catalog — Production Patterns and Pitfalls

You've got terabytes of raw logs in S3, a dozen CSV exports from legacy systems, and a mandate to build a unified analytics view. The old way: spin up an EMR cluster, write Spark scripts, manage YARN, and pray the spot instances don't get reclaimed. That's what AWS Glue was built to kill — but it's not magic. I've seen Glue jobs silently fail at 3 AM because of a single corrupt Parquet file, and I've watched teams burn $10k on idle DPUs because they didn't understand auto-scaling. This isn't a tutorial. It's a field guide to making Glue work in production without getting burned. By the end, you'll know how to design robust Glue ETL pipelines, tune performance, handle failures, and avoid the traps that catch everyone.

Why Glue Exists: The Pain Before Serverless ETL

Before Glue, every ETL pipeline meant managing infrastructure. You'd launch an EMR cluster, configure Spark, handle spot instance interruptions, and write custom scripts for schema discovery. The Data Catalog didn't exist — you either hardcoded schemas or used Hive Metastore, which required its own servers. Glue solves two problems: 1) automatically cataloging data sources so you don't manually track schemas, and 2) running Spark jobs without provisioning clusters. But the trade-off is less control over execution environment and Spark config. You can't install custom libraries easily, and debugging is harder because you don't have direct SSH access to executors. For teams that value speed over fine-grained control, Glue is a win. For complex, latency-sensitive pipelines, consider EMR or Spark on Kubernetes.

# io.thecodeforge — DevOps tutorial

import boto3

glue = boto3.client('glue')

# Create a crawler for S3 data
response = glue.create_crawler(
    Name='s3-logs-crawler',
    Role='arn:aws:iam::123456789012:role/GlueServiceRole',
    DatabaseName='logs_db',
    Targets={'S3Targets': [{'Path': 's3://my-bucket/logs/'}]},
    SchemaChangePolicy={'UpdateBehavior': 'UPDATE_IN_DATABASE', 'DeleteBehavior': 'LOG'},
    Configuration='{"Version":1.0,"CrawlerOutput":{"Partitions":{"AddOrUpdateBehavior":"InheritFromTable"}}}'
)

# Start the crawler
glue.start_crawler(Name='s3-logs-crawler')
print("Crawler started. Check CloudWatch for progress.")

Glue ETL Jobs: Anatomy of a Spark Job on Serverless

A Glue job is essentially a Spark application running on an ephemeral cluster. You write Python (PySpark) or Scala code, and Glue handles cluster provisioning, scaling, and teardown. The key differentiator from plain Spark is the Glue DynamicFrame — a distributed collection of records that supports schema inference and transformations like ResolveChoice (handles schema inconsistencies) and Relationalize (flattens nested data). Under the hood, DynamicFrames convert to Spark DataFrames for execution. The job runs on a cluster of worker nodes, each with a fixed number of DPUs (Data Processing Units). One DPU equals 4 vCPU and 16 GB memory. You choose the number of workers and worker type (Standard, G.1x, G.2x) based on data size and complexity. The job bookmarks track processed data to avoid reprocessing — critical for incremental loads.

# io.thecodeforge — DevOps tutorial

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'S3_INPUT_PATH', 'S3_OUTPUT_PATH'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Read from Data Catalog table (crawled earlier)
dynamic_frame = glueContext.create_dynamic_frame.from_catalog(
    database="logs_db",
    table_name="raw_logs",
    transformation_ctx="raw_logs"
)

# Resolve schema conflicts: choose first type encountered
resolved = ResolveChoice.apply(frame=dynamic_frame, choice="make_cols", transformation_ctx="resolved")

# Filter out null user agents
filtered = Filter.apply(frame=resolved, f=lambda row: row["user_agent"] is not None, transformation_ctx="filtered")

# Write as Parquet with Snappy compression
glueContext.write_dynamic_frame.from_options(
    frame=filtered,
    connection_type="s3",
    connection_options={"path": args['S3_OUTPUT_PATH'], "partitionKeys": ["event_date"]},
    format="parquet",
    format_options={"compression": "snappy"},
    transformation_ctx="output"
)

job.commit()

Data Catalog: The Metadata Backbone

The Data Catalog is a managed Hive Metastore compatible with Athena, Redshift Spectrum, and EMR. It stores table definitions, partition metadata, and schema versions. Crawlers automatically populate it by scanning data sources. Each table has a location (S3 path), format (Parquet, CSV, etc.), and serde (serialization/deserialization) library. Partitions are stored as Hive-style directories (e.g., event_date=2025-01-01/). The Catalog supports schema versioning — you can view changes over time. Critical for production: the Catalog is eventually consistent. After a crawler updates a table, there's a propagation delay (seconds to minutes). If your job reads immediately after a crawler, it might see stale metadata. Always add a retry or delay. Also, the Catalog has a default limit of 100,000 tables per account — request a limit increase if you have many data sources.

# io.thecodeforge — DevOps tutorial

import boto3

glue = boto3.client('glue')

# Get table metadata
table = glue.get_table(DatabaseName='logs_db', Name='raw_logs')
print("Table location:", table['Table']['StorageDescriptor']['Location'])
print("Columns:")
for col in table['Table']['StorageDescriptor']['Columns']:
    print(f"  {col['Name']}: {col['Type']}")

# List partitions
partitions = glue.get_partitions(DatabaseName='logs_db', TableName='raw_logs')
print(f"Number of partitions: {len(partitions['Partitions'])}")

Performance Tuning: DPUs, Workers, and Spark Config

Glue jobs run on worker types: Standard (1 DPU per worker), G.1x (4 DPU), G.2x (8 DPU). More DPUs means more parallelism, but also more shuffle overhead. The default max concurrency per job is 1 — you can increase it for parallel runs. Key Spark configs: spark.executor.memory (default 4GB for Standard), spark.driver.memory (default 4GB), spark.sql.shuffle.partitions (default 200). For large datasets, increase shuffle partitions to avoid OOM. Use spark.sql.adaptive.enabled=true for dynamic coalescing. Glue also supports auto-scaling — set --enable-auto-scaling=true. This adds workers up to the max you specify. But auto-scaling adds latency (2-5 min to scale up). For predictable workloads, set a fixed worker count. Monitor CloudWatch metrics: DPUUsage, JobRunTime, and ShuffleBytesWritten. If shuffle is high, increase parallelism or use bucketing.

# io.thecodeforge — DevOps tutorial

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'S3_INPUT_PATH', 'S3_OUTPUT_PATH'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Set Spark config for performance
spark.conf.set("spark.sql.shuffle.partitions", "400")
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")

df = spark.read.parquet(args['S3_INPUT_PATH'])
# Perform transformations...
df.write.mode("overwrite").parquet(args['S3_OUTPUT_PATH'])

job.commit()

Handling Failures: Retries, Alarms, and Dead Letter Queues

Glue jobs fail for many reasons: corrupt data, OOM, network timeouts, IAM permission changes. By default, Glue retries failed jobs up to 3 times with exponential backoff. But retries don't help if the root cause is a bad file. Implement a dead letter queue (DLQ) pattern: move failed records to an S3 bucket for manual inspection. Use Glue's job bookmark to skip already processed files after a retry. Set up CloudWatch alarms on JobRunState (FAILED) and DPUUsage (spikes). For critical pipelines, use AWS Step Functions to orchestrate Glue jobs with custom error handling (e.g., send SNS notification, pause pipeline). Also, enable continuous logging to CloudWatch (--enable-continuous-cloudwatch-log=true) to capture Spark driver and executor logs. Without it, you only get a summary — useless for debugging.

# io.thecodeforge — DevOps tutorial

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'S3_INPUT_PATH', 'S3_OUTPUT_PATH', 'S3_DLQ_PATH'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

df = spark.read.parquet(args['S3_INPUT_PATH'])

# Separate valid and invalid records
try:
    valid_df = df.filter(df["status_code"].isNotNull())
    invalid_df = df.filter(df["status_code"].isNull())
    
    # Write valid
    valid_df.write.mode("append").parquet(args['S3_OUTPUT_PATH'])
    
    # Write invalid to DLQ
    if invalid_df.count() > 0:
        invalid_df.write.mode("append").parquet(args['S3_DLQ_PATH'])
        print(f"Moved {invalid_df.count()} invalid records to DLQ")
except Exception as e:
    print(f"Job failed: {e}")
    raise

job.commit()

Security: IAM Roles, Encryption, and VPC

Glue jobs run with an IAM role that needs permissions to read/write S3, access the Data Catalog, and optionally connect to JDBC sources. The role must have a trust policy for glue.amazonaws.com. For sensitive data, enable S3 server-side encryption (SSE-S3 or SSE-KMS) and use Glue's built-in encryption for the Data Catalog (set at catalog level). If your data is in a VPC (e.g., RDS, Redshift), you must attach a VPC to the Glue job. This adds a network interface per worker and can slow down S3 access. Use VPC endpoints for S3 and DynamoDB to avoid NAT gateway costs. Also, Glue supports AWS Lake Formation for fine-grained access control — you can restrict column-level access. But Lake Formation adds complexity; only use it if you need multi-team data sharing.

# io.thecodeforge — DevOps tutorial

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'JDBC_URL', 'DB_USER', 'DB_PASSWORD'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Read from RDS via JDBC (requires VPC)
df = spark.read.format("jdbc").options(
    url=args['JDBC_URL'],
    dbtable="orders",
    user=args['DB_USER'],
    password=args['DB_PASSWORD'],
    driver="com.mysql.cj.jdbc.Driver"
).load()

# Transform and write to S3
df.write.mode("overwrite").parquet("s3://my-bucket/orders/")

job.commit()

Monitoring and Observability: Beyond CloudWatch

CloudWatch logs are the default, but they're noisy. Use structured logging (JSON) to filter easily. Glue emits metrics: DPUUsage, JobRunTime, BytesRead, BytesWritten, ShuffleBytesWritten. Set up dashboards for these. For Spark-specific metrics, enable the Spark UI (--enable-spark-ui=true) and view it via the Glue console or proxy. The Spark UI shows stages, tasks, and shuffle details — invaluable for tuning. Also, use Glue's job run insights (available in the console) to see bottlenecks. For alerting, create CloudWatch alarms on JobRunState (FAILED) and DPUUsage (if it drops unexpectedly, the job may be idle). Consider using AWS Distro for OpenTelemetry to send traces to X-Ray for end-to-end visibility.

# io.thecodeforge — DevOps tutorial

import sys
import json
import logging
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

# Configure structured logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Log with JSON
log_data = {"event": "job_start", "job_name": args['JOB_NAME'], "timestamp": spark.sparkContext.startTime}
logger.info(json.dumps(log_data))

# ... job logic ...

job.commit()
log_data = {"event": "job_end", "job_name": args['JOB_NAME'], "status": "success"}
logger.info(json.dumps(log_data))

When Not to Use Glue: Alternatives and Trade-offs

Glue is not a silver bullet. Avoid it when: 1) You need sub-minute latency — Glue jobs have a startup overhead of 1-5 minutes. Use AWS Lambda or Kinesis Data Analytics for streaming. 2) You need custom Spark libraries (e.g., MLlib with native dependencies) — Glue's environment is locked down. Use EMR or SageMaker. 3) You have complex multi-step workflows with branching — Glue jobs are single scripts. Use Step Functions to orchestrate multiple Glue jobs, but that adds complexity. 4) You need fine-grained cost control — Glue charges per DPU-hour regardless of utilization. For predictable workloads, reserved capacity (EMR) may be cheaper. 5) You need to debug with SSH — Glue doesn't allow direct access to executors. Use EMR if you need deep debugging. The rule of thumb: if your ETL fits in a single Spark script and you don't need custom environments, Glue is great. Otherwise, consider EMR, Athena, or custom Spark on Kubernetes.