Data Lake vs Lakehouse: Delta Lake, Iceberg, Hudi Deep Dive

In the early days of big data, organizations built data lakes to store massive amounts of raw data cheaply on object storage like Amazon S3 or HDFS. However, data lakes suffered from a lack of ACID transactions, making it difficult to ensure data consistency, especially when multiple jobs write concurrently. They also lacked schema enforcement, leading to data corruption over time. The lakehouse architecture emerged as a solution, combining the low-cost storage of data lakes with the reliability and performance of data warehouses. At the heart of lakehouses are open-source table formats: Delta Lake, Apache Iceberg, and Apache Hudi. These formats add transactional metadata layers on top of Parquet files, enabling ACID transactions, schema evolution, time travel, and efficient upserts. This tutorial will dive deep into each format, compare their strengths, and provide production-ready examples. By the end, you'll understand how to choose the right format for your use case and debug common issues in production.

What is a Data Lake?

A data lake is a centralized repository that stores all your structured and unstructured data at any scale. You can store data as-is, without having to first structure it, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning. Data lakes are typically built on low-cost object storage like Amazon S3, Azure Data Lake Storage, or HDFS. The key advantage is cost and flexibility. However, traditional data lakes lack ACID transactions, making it hard to ensure data consistency when multiple jobs write concurrently. They also lack schema enforcement, leading to data quality issues. For example, if a streaming job writes to a table while an analytics query reads it, the query might see partial writes or inconsistent data. This is where lakehouse formats come in.

-- Simulating a data lake write without ACID
-- This is a Spark SQL example on raw Parquet files
CREATE TABLE sales_data USING parquet LOCATION '/data/sales';
INSERT INTO sales_data VALUES (1, '2023-01-01', 100);
-- Another job overwrites the table
INSERT OVERWRITE sales_data VALUES (2, '2023-01-02', 200);
-- Query might see partial results if concurrent
SELECT * FROM sales_data;

What is a Lakehouse?

A lakehouse is a new, open architecture that combines the best elements of data lakes and data warehouses. It is built on low-cost object storage and uses a transactional metadata layer to provide ACID transactions, schema enforcement, and data versioning. This enables reliable data processing, including streaming and batch, with the performance of a warehouse. The core components are table formats like Delta Lake, Apache Iceberg, and Apache Hudi. These formats add a transaction log that records every change, allowing for snapshot isolation, time travel, and rollbacks. They also support schema evolution, so you can add or rename columns without rewriting the entire table. Lakehouses are ideal for modern data platforms that need to support BI, ML, and real-time analytics on a single copy of data.

-- Creating a Delta Lake table with ACID
CREATE TABLE sales_delta (
  id INT,
  date DATE,
  amount DECIMAL(10,2)
) USING delta LOCATION '/data/sales_delta';

INSERT INTO sales_delta VALUES (1, '2023-01-01', 100);
-- Concurrent insert
INSERT INTO sales_delta VALUES (2, '2023-01-02', 200);
-- Query sees consistent snapshot
SELECT * FROM sales_delta;

-- Time travel: see previous version
SELECT * FROM sales_delta VERSION AS OF 0;

Delta Lake: The Pioneer

Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark and big data workloads. It was developed by Databricks and is now part of the Linux Foundation. Delta Lake uses a transaction log (stored as JSON files in a _delta_log directory) to record every change. It supports schema enforcement, schema evolution, time travel, and unified batch/streaming. Delta Lake is tightly integrated with Spark, but also supports Presto, Trino, and Flink through connectors. Key features include: ACID transactions with serializable isolation, scalable metadata handling, and built-in data compaction (OPTIMIZE). Delta Lake is a good choice if your ecosystem is Spark-centric.

-- Delta Lake specific operations
-- Convert existing Parquet to Delta
CONVERT TO DELTA parquet.`/data/sales`;

-- Schema evolution: add column
ALTER TABLE sales_delta ADD COLUMNS (region STRING);

-- Merge (upsert)
MERGE INTO sales_delta AS target
USING updates AS source
ON target.id = source.id
WHEN MATCHED THEN UPDATE SET target.amount = source.amount
WHEN NOT MATCHED THEN INSERT *;

-- Optimize (compact small files)
OPTIMIZE sales_delta;

-- Vacuum (clean up old files)
VACUUM sales_delta RETAIN 168 HOURS;

Apache Iceberg: The Multi-Engine Champion

Apache Iceberg is an open table format designed for huge analytic datasets. It was developed at Netflix and is now an Apache top-level project. Iceberg's key strength is its engine-agnostic design: it works with Spark, Flink, Trino, Presto, Hive, and more. Iceberg uses a tree structure of metadata files (metadata.json, manifest lists, manifests) to track table snapshots. It supports ACID transactions with serializable isolation, schema evolution, hidden partitioning (partitioning is defined at table creation and automatically applied), and time travel. Iceberg's partition evolution allows you to change partitioning without rewriting data. It also supports row-level delete files for efficient updates and deletes. Iceberg is ideal for multi-engine environments.

-- Creating an Iceberg table
CREATE TABLE sales_iceberg (
  id INT,
  date DATE,
  amount DECIMAL(10,2)
) USING iceberg PARTITIONED BY (date);

-- Insert data
INSERT INTO sales_iceberg VALUES (1, '2023-01-01', 100);

-- Time travel
SELECT * FROM sales_iceberg FOR SYSTEM_TIME AS OF '2023-01-01 00:00:00';

-- Schema evolution
ALTER TABLE sales_iceberg ADD COLUMN region STRING;

-- Partition evolution (change partition spec)
ALTER TABLE sales_iceberg SET PARTITION SPEC (bucket(16, id));

-- Expire snapshots
CALL iceberg.system.expire_snapshots('default.sales_iceberg', TIMESTAMP '2023-01-02 00:00:00');

Apache Hudi: The Near-Real-Time Upsert Specialist

Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data management framework that provides upserts, incremental queries, and data compaction. It was developed at Uber and is now an Apache project. Hudi is designed for near-real-time data ingestion with low latency. It supports two table types: Copy-on-Write (CoW) and Merge-on-Read (MoR). CoW stores data in columnar files (Parquet) and rewrites files on upsert, while MoR uses a combination of columnar and row-based (Avro) files for faster writes. Hudi provides ACID transactions, time travel, and incremental queries (only new changes). It integrates with Spark, Flink, Presto, and Trino. Hudi is ideal for streaming ingestion with frequent upserts.

-- Creating a Hudi table (Copy-on-Write)
CREATE TABLE sales_hudi (
  id INT,
  date DATE,
  amount DECIMAL(10,2),
  PRIMARY KEY (id) NOT ENFORCED
) USING hudi
OPTIONS (
  type = 'cow',
  precombine.field = 'date'
) PARTITIONED BY (date);

-- Insert data
INSERT INTO sales_hudi VALUES (1, '2023-01-01', 100);

-- Upsert (using merge)
MERGE INTO sales_hudi AS target
USING updates AS source
ON target.id = source.id
WHEN MATCHED THEN UPDATE SET target.amount = source.amount
WHEN NOT MATCHED THEN INSERT *;

-- Incremental query (get changes after a commit)
SELECT * FROM sales_hudi WHERE _hoodie_commit_time > '20230101000000';

-- Compaction (for MoR tables)
CALL run_compaction(table => 'sales_hudi', op => 'run');

Comparison: Delta Lake vs Iceberg vs Hudi

Choosing the right format depends on your use case. Here's a comparison: Delta Lake is best for Spark-centric environments with heavy batch processing; Iceberg is best for multi-engine environments with complex schemas; Hudi is best for near-real-time upserts and incremental processing. All three support ACID, time travel, and schema evolution. Delta Lake has the most mature ecosystem with Databricks support. Iceberg has the best engine compatibility. Hudi has the best upsert performance. In terms of metadata management, Iceberg uses a tree structure that scales well, while Delta Lake's transaction log can become a bottleneck. Hudi's metadata is more complex due to log files. For compaction, Delta Lake's OPTIMIZE is straightforward, Iceberg requires snapshot expiration, and Hudi requires separate compaction runs.

-- Example of converting between formats (conceptual)
-- Delta to Iceberg: use Apache Spark with Iceberg connector
-- Iceberg to Hudi: rewrite using Hudi's bulk_insert
-- No direct conversion; requires rewriting data

Production Best Practices

Running lakehouse formats in production requires careful configuration. For Delta Lake, set 'delta.enableChangeDataFeed = true' for CDC, and use 'delta.autoOptimize.autoCompact = true' to automatically compact small files. For Iceberg, configure 'write.metadata.metrics.default' to control column statistics, and set 'write.target-file-size-bytes' to avoid small files. For Hudi, tune 'hoodie.upsert.shuffle.parallelism' and 'hoodie.cleaner.policy' to manage file sizes. Always monitor the transaction log size and metadata overhead. Use partitioning wisely to avoid too many partitions (e.g., daily partitioning for date columns). Implement data retention policies using VACUUM (Delta), expire snapshots (Iceberg), or cleaner (Hudi). Test time travel and rollback procedures regularly.

-- Delta Lake: enable auto compaction
ALTER TABLE sales_delta SET TBLPROPERTIES (
  'delta.autoOptimize.autoCompact' = 'true',
  'delta.targetFileSize' = '256mb'
);

-- Iceberg: set target file size
ALTER TABLE sales_iceberg SET TBLPROPERTIES (
  'write.target-file-size-bytes' = '268435456'
);

-- Hudi: tune parallelism
SET hoodie.upsert.shuffle.parallelism = 200;