Lakehouse Guide

TL;DR

• A lakehouse combines lake economics with warehouse semantics — cheap object storage (S3, ADLS, GCS) plus ACID transactions, schema enforcement, and fast SQL engines on the same files.
• Open table formats are the enabling layer — Delta Lake, Apache Iceberg, and Apache Hudi add transactional metadata, versioning, and time travel on top of Parquet/ORC files.
• You stop maintaining two copies of truth — no more separate ETL from raw data lake to curated data warehouse for every dataset; bronze/silver/gold layers live on the same storage with different table policies.
• Compute is interchangeable — Spark, Trino, Flink, DuckDB, and even Snowflake external tables read and write the same governed tables through a shared catalog.
• The hard parts are governance, compaction, and format choice — not spinning up Spark. Production lakehouses fail on orphaned small files, schema drift, and teams writing incompatible formats side by side.

Quick Decision Guide

Who this guide is for

• Best for: data engineers · analytics engineers · platform engineers · ML engineers · architects
• Difficulty: Intermediate
• Estimated time: 50 min

Learning Path

ETL → Data Lakes → Data Warehouses → Lakehouse → AI System Architecture → Embeddings

On this page

• Why This Matters
• The Problem Lakehouses Solve
• How We Got Here
• What Is a Lakehouse?
• How Lakehouses Work
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Use a Lakehouse
• Running in Production
• Production Checklist
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References
• Further Reading

Why This Matters

For a decade, data teams ran two systems: a data lake for cheap storage of raw JSON, logs, and events, and a data warehouse for reliable SQL, BI, and SLA-backed dashboards. Keeping them in sync meant duplicate ETL pipelines, stale warehouse copies, and angry analysts when the lake had fresher data than Snowflake.

Lakehouse architecture collapses that split. Raw and curated data live on object storage. Open table formats provide transactions, schema evolution, and partition pruning so Spark, Trino, DuckDB, and warehouse engines query the same files with predictable behavior. The storage layer becomes open and durable; compute becomes a replaceable commodity you attach to it.

If you build AI products, lakehouses matter twice: training datasets, feature tables, and document metadata for embedding pipelines originate in the lake, while product analytics still need warehouse-grade correctness. One governed storage layer reduces cost, drift, and the number of places a compliance audit has to look. See AI System Architecture for how the lakehouse sits under retrieval and ML serving layers.

Engineering Insight

A lakehouse is not a product you buy — it is a contract between storage and compute. The open table format is the architecture; everything else (Spark, Trino, Databricks, Snowflake) is an interchangeable client of that contract.

The Problem Lakehouses Solve

The two-system world created structural problems that no amount of pipeline discipline could fix:

Duplicate truth. Every dataset that mattered existed twice — once as raw files in the lake, once as loaded tables in the warehouse. Each copy had its own freshness, its own permissions model, and its own bill. When they disagreed, whole afternoons went to reconciliation.

No transactions on the lake. Plain Parquet files on S3 have no commit protocol. A failed Spark job left half-written partitions that downstream readers happily consumed. Concurrent writers corrupted each other. "Rerun the pipeline and hope" was the recovery strategy.

Warehouse economics punish raw data. Loading petabytes of clickstream into a warehouse just so analysts might query it is expensive. So raw data stayed in the lake — unqueryable by the BI team — and the warehouse held only what someone had already decided was important.

ML teams routed around governance. Data scientists needed raw events and full history, which the warehouse didn't hold, so they read the lake directly — outside row-level security, outside audit logs, outside lineage.

Lakehouses do not eliminate warehouses entirely. Snowflake, BigQuery, and Redshift increasingly read Iceberg/Delta tables externally. The architectural shift is: storage and table semantics are open; compute is interchangeable.

How We Got Here

The lakehouse is the third act of a twenty-year story about where analytical data lives.

Diagram: From Hadoop lakes to open table format convergence

timeline
    title Evolution of the lakehouse
    2006-2012 : Hadoop + HDFS era
              : Hive adds SQL-on-files
    2013-2016 : Cloud object storage lakes
              : S3 + Parquet, no transactions
    2017-2019 : Table formats emerge
              : Hudi (Uber), Iceberg (Netflix), Delta (Databricks)
    2020 : Databricks coins "lakehouse"
         : CIDR paper formalizes the pattern
    2021-2023 : Format war heats up
              : Snowflake, AWS, Google adopt Iceberg
    2024-2026 : Convergence and interop
              : Delta UniForm, Iceberg REST catalogs, Polaris

Table formats were invented independently at Uber, Netflix, and Databricks to fix the same problem: no ACID on object storage.

Three independent teams hit the same wall around 2017. Uber built Hudi because CDC upserts into HDFS were unmanageable. Netflix built Iceberg because Hive's directory-based table layout broke at their scale — a single partition rename could take hours and corrupt readers. Databricks built Delta Lake to give Spark pipelines transactional guarantees. In 2020 Databricks published the CIDR paper "Lakehouse: A New Generation of Open Platforms," giving the pattern its name. By 2023–2025, Iceberg adoption by Snowflake, AWS Athena/Glue, and Google BigLake — plus Databricks acquiring Tabular (founded by Iceberg's creators) — signaled convergence: the format layer is open, and every vendor competes on compute and governance above it.

What Is a Lakehouse?

A lakehouse is an architectural pattern where:

1. Storage is object storage (S3-compatible) holding open file formats — typically Parquet, sometimes ORC or Avro.
2. Table layer adds database-like guarantees via an open table format: ACID commits, snapshots, schema evolution, time travel.
3. Compute engines (Spark, Flink, Trino, Databricks SQL, DuckDB, Snowflake external tables) read and write through that table layer.
4. Catalog (Hive Metastore, Unity Catalog, AWS Glue, Apache Polaris) tracks table locations, schemas, and permissions.

The name was popularized by Databricks, but the idea is vendor-neutral: decouple storage from compute while keeping warehouse features. A table stops being "a directory of files" and becomes "a metadata pointer to a consistent snapshot of files" — that single change is what makes transactions, concurrent writers, and time travel possible on dumb object storage.

What distinguishes a lakehouse from adjacent patterns:

How Lakehouses Work

Everything reduces to two paths — how data gets in, and how queries get it out — with the table format's metadata mediating both.

Write path

An engine writes new immutable data files (Parquet) to object storage, then atomically commits a new metadata snapshot that references them. The commit is a single atomic operation — a compare-and-swap on the catalog pointer (Iceberg) or an ordered JSON entry in the transaction log (Delta's _delta_log). Readers never see partial writes: until the commit lands, the new files are invisible. On failure, uncommitted files are orphaned and later garbage-collected. Concurrent writers use optimistic concurrency — both prepare files, one commit wins, the loser retries against the new snapshot.

Read path

The engine resolves the table through the catalog, reads the latest snapshot (or a historical one for time travel), and plans the scan using metadata alone: partition values, per-file column statistics (min/max, null counts), and file sizes. Files that cannot contain matching rows are pruned before a single byte of data is read. Only then does the engine fetch the relevant Parquet files — and within them, only the projected columns.

This is why lakehouse performance is dominated by metadata hygiene: good statistics, right-sized files, and sensible partitioning let the planner skip 99% of a petabyte table.

Open table formats

The three production formats implement the same core ideas with different mechanics:

• Delta Lake — a JSON transaction log (_delta_log/) plus periodic Parquet checkpoints. Every commit appends an ordered log entry. Deep Spark and Databricks integration; Unity Catalog provides enterprise governance; UniForm exposes Delta tables with Iceberg-compatible metadata.
• Apache Iceberg — a spec-first metadata tree: table metadata file → manifest lists → manifests → data files. Engine-neutral by design; hidden partitioning and partition evolution let you change partition schemes without rewriting data or breaking queries. See Iceberg.
• Apache Hudi — record-level indexing with copy-on-write and merge-on-read table types, optimized for streaming upserts and CDC ingestion from operational databases.

All three provide ACID commits, schema evolution, and time travel. The differences that matter in production are engine ecosystem, partition management, and upsert mechanics — covered in Comparisons.

Architecture

The dominant physical layout is the medallion architecture: three quality tiers of tables on the same object storage, all in the same table format, governed by the same catalog.

Diagram: Medallion lakehouse on object storage

flowchart LR
    subgraph Sources
        APP[App events]
        DB[OLTP CDC]
        API[SaaS APIs]
    end
    subgraph OS["Object storage (S3 / ADLS / GCS)"]
        B[(Bronze<br/>raw tables)]
        S[(Silver<br/>cleaned tables)]
        G[(Gold<br/>metrics tables)]
    end
    CAT[Catalog<br/>schemas + ACLs]
    subgraph Consumers
        BI[BI / dashboards]
        ML[ML features]
        EMB[Embedding pipelines]
    end
    APP --> B
    DB --> B
    API --> B
    B -->|dbt / Spark| S
    S -->|dbt / Spark| G
    CAT -.governs.-> B
    CAT -.governs.-> S
    CAT -.governs.-> G
    G --> BI
    S --> ML
    S --> EMB

All three layers are tables in one open format on one bucket — only policies (retention, contracts, access) differ per layer.

Around the storage core sit four supporting planes:

• Ingestion: batch ETL jobs (Spark, Airbyte) and streaming writers (Flink, Kafka Connect, Spark Structured Streaming) landing into bronze.
• Transformation: dbt or Spark jobs promoting bronze → silver → gold, with tests as promotion gates.
• Query: Trino for interactive SQL, Spark for heavy transforms, DuckDB for local/dev analysis, warehouse engines via external tables.
• Governance: the catalog enforces table ACLs, row/column policies, lineage, and PII classification — the piece that makes ML access safe instead of a bypass.

Step-by-Step Flow

The atomic commit is the heart of the lakehouse. Here is what one write actually does, end to end.

Diagram: Write commit and concurrent read (Iceberg-style)

sequenceDiagram
    participant W as Writer (Spark job)
    participant OS as Object storage
    participant C as Catalog
    participant R as Reader (Trino)
    W->>OS: 1. Write new Parquet data files
    W->>OS: 2. Write manifests + snapshot metadata
    W->>C: 3. Commit: swap table pointer (CAS, expects v41)
    alt Pointer still at v41
        C-->>W: Commit OK → table now v42
    else Another writer won
        C-->>W: Conflict → re-plan against v42, retry
    end
    R->>C: 4. Resolve table → current snapshot (v42)
    R->>OS: 5. Read metadata, prune files by stats
    R->>OS: 6. Scan only surviving Parquet files

Data files land first and are invisible; the single catalog pointer swap is the only atomic step, which is why readers never see partial writes.

Walking through it:

1. Stage data files. The writer produces new immutable Parquet files in the table's location. Nothing references them yet — a crash here leaves harmless orphans.
2. Stage metadata. The writer builds manifests listing the new files with per-column statistics, and a new snapshot that includes prior data plus the new files (or replaces files, for updates/deletes).
3. Atomic commit. The writer asks the catalog to swap the table's current-metadata pointer, conditioned on the version it started from. This compare-and-swap either fully succeeds or fully fails — the ACID guarantee lives in this one step.
4. Conflict handling. If a concurrent writer committed first, the swap fails. The writer re-checks whether its changes still apply (append vs conflicting rewrite), rebases, and retries.
5. Reads resolve through the catalog. A reader always gets a complete, consistent snapshot — the latest one, or a pinned historical snapshot for time travel and reproducible ML training runs.
6. Prune, then scan. Partition values and file statistics eliminate most files before any data I/O happens.

Delta Lake implements the same protocol with an ordered log: committing means atomically creating _delta_log/00000000000000000042.json; whoever creates the next log entry first wins.

Real Production Example

A fintech company ingests 2 TB/day of transaction and app events into bronze Iceberg tables on S3. Flink streaming jobs deduplicate and conform events into silver tables within 15 minutes of arrival. dbt running on Spark builds gold metrics tables nightly. Trino serves analyst SQL against silver and gold; Snowflake reads gold via an external Iceberg catalog for executive dashboards — no copy, no sync pipeline.

The ML team materializes silver transaction features into a feature store nightly, pinning the Iceberg snapshot ID per training run so experiments are reproducible. Document embedding pipelines for the compliance search product read document metadata (owner, effective date, jurisdiction) from a gold table, so the retrieval layer inherits the same governance as BI.

Time travel earns its keep during audits. When a regulator asks what customer risk scores looked like on a specific date, the answer is a query, not an archaeology project:

-- Audit: what did the customers table look like yesterday?
SELECT customer_id, risk_score, updated_at
FROM prod.silver.customers
FOR TIMESTAMP AS OF TIMESTAMP '2026-07-20 00:00:00 UTC'
WHERE country = 'US';

-- Reproduce a training set: pin the exact snapshot a model was trained on
SELECT *
FROM prod.silver.transaction_features
FOR VERSION AS OF 8231648994821754321;

Operational numbers that made this work: file compaction targets 256 MB, snapshot retention is 7 days on bronze and 30 days on silver/gold (compliance tables keep 7 years via branch tags), and a weekly expire_snapshots + orphan-file cleanup keeps storage growth linear with data instead of with write frequency.

The incident that validated the design: a bad deploy wrote corrupted amounts into silver for 40 minutes. Rollback was CALL rollback_to_snapshot(...) — one metadata operation, zero data movement — followed by replaying the window from bronze.

Design Decisions

The decisions that shape a lakehouse build, in the order you should make them:

Three of these deserve emphasis:

Format is a one-way door in practice. Migrating petabytes between formats means rewrite jobs, dual-write windows, and coordinating every reader. Standardize per data domain, document the exception process, and enforce it at write time.

The catalog is the real control plane. Access policies, lineage, and discovery all hang off it. A lakehouse with a weak catalog is a data swamp with transactions.

Partitioning strategy determines query cost. Partition by what queries filter on (usually event date, sometimes tenant). Iceberg's hidden partitioning removes the classic failure where analysts forget the dt= predicate and full-scan the table.

Common patterns

• Streaming medallion: Kafka → Flink → bronze (append) → silver (MoR upserts) → gold (batch aggregates).
• Warehouse offload: keep Snowflake/BigQuery for BI UX, but point them at Iceberg external tables so storage is open and paid once.
• ML snapshot pinning: every training job records the table snapshot IDs it read; retraining and audits replay exactly.
• Branch-based backfills (Iceberg): write a backfill to a table branch, validate with dbt tests, then fast-forward the main branch — the lakehouse version of a staging deploy.

Comparisons

Lake vs warehouse vs lakehouse

Delta Lake vs Iceberg vs Hudi

Decision tree: choosing a table format

flowchart TD
    A{Primary platform?} -->|Databricks / Spark-centric| B[Delta Lake]
    A -->|Multi-engine or undecided| C{Workload shape?}
    C -->|High-churn CDC upserts| D[Hudi]
    C -->|Analytics + mixed engines| E[Iceberg]
    B --> F{Need non-Spark readers?}
    F -->|Yes| G[Delta + UniForm<br/>Iceberg-compatible metadata]
    F -->|No| H[Plain Delta]
    E --> I{Warehouse must read it?}
    I -->|Yes| J[Iceberg + REST catalog<br/>Snowflake/BigQuery external]
    I -->|No| K[Iceberg + Glue/Polaris]

Platform gravity decides most format choices; workload shape (CDC vs analytics) decides the rest.

Important

Format wars are real, but interop is improving: Delta UniForm and Apache XTable expose one physical table through multiple format APIs. Treat interop as a migration bridge, not a license to let every team pick its own format.

Common Mistakes

1. Millions of tiny files. Streaming micro-batches without compaction destroy query performance — planning time explodes and scan parallelism collapses. Schedule compaction (OPTIMIZE, rewrite_data_files) from day one, targeting 128–512 MB files.
2. No partition strategy — or the wrong one. Full scans on petabyte tables are expensive; partitioning by high-cardinality columns (user ID) creates the small-files problem at directory scale. Partition by date or tenant; use Iceberg hidden partitioning where helpful.
3. Schema chaos in bronze leaking downstream. Allow evolution in bronze, but gate silver with contracts — dbt tests or Great Expectations — so a producer adding a column doesn't silently break every consumer.
4. Mixing formats in one pipeline. Delta writers and Iceberg readers pointed at the same logical dataset is a support nightmare. One format per domain, enforced at write time.
5. Ignoring snapshot retention. Time travel is stored history: every snapshot pins files. Without VACUUM/expire_snapshots, storage grows with write frequency, not data volume.
6. Treating the lakehouse as a dump. Without catalog ownership, lineage, and PII classification, you recreate the data swamp with better marketing.
7. MERGE-heavy patterns for OLTP-shaped workloads. Row-at-a-time upserts at high QPS belong in an operational database; lakehouse MERGE is built for batches.
8. Skipping statistics maintenance. Stale or missing column stats defeat file pruning — the planner reads everything and the "fast" lakehouse mysteriously isn't.

Where It Breaks Down

Lakehouses excel at analytical and batch/streaming ML workloads. Honest failure zones:

• Commit contention. Optimistic concurrency degrades when many writers hit one table: retries pile up, latency spikes. Hundreds of concurrent streaming writers per table need careful design (partitioned writers, MoR) or a different architecture.
• Small-batch streaming economics. Second-level freshness means tiny commits, metadata churn, and constant compaction. Below ~1-minute latency requirements, a streaming system (Kafka + Flink state, or a real-time OLAP store) in front of the lakehouse works better.
• Metadata scale. Tables with millions of files or tens of thousands of snapshots slow planning itself. Compaction, manifest rewriting, and snapshot expiry become mandatory operational jobs, not nice-to-haves.
• Point lookups and serving. Fetching one row by key is object-storage latency (tens to hundreds of ms) plus planning. Serving layers — feature stores, vector databases for retrieval, Redis — should sync from the lakehouse, not query it per request.
• Unstructured blobs. Table formats govern tabular metadata, not PDF or image content. Store blobs in object storage, index their metadata in lakehouse tables, and process content with embedding pipelines.

When NOT to Use a Lakehouse

The pattern in every row: the lakehouse is a system of record and heavy-compute substrate. When the requirement is latency (serving) or simplicity (small scale), attach a specialized layer or skip the lakehouse entirely until scale forces the issue.

Running in Production

Important

A lakehouse without compaction, catalog discipline, and access policies is an expensive data swamp with better marketing. The table format gives you correctness; operations give you performance and trust.

Freshness SLOs by layer keep expectations honest: bronze within minutes of the source, silver within the transformation cadence (15 min–1 hr), gold per business agreement (often hourly/daily). Publish them; alert on breach, not on job failure — a job can fail and retry within SLO.

Upgrades are fleet operations. Format spec versions (Iceberg v2 → v3, Delta protocol versions) gate features like deletion vectors. Upgrading a table can break old readers, so inventory every engine version that touches a table before flipping protocol flags.

Continue Learning

• Next guide: AI System Architecture
• Then: Embeddings

Production Checklist

• One table format per domain, documented, enforced at write time
• Compaction jobs scheduled; target file size documented per table
• Snapshot retention + orphan file cleanup automated per layer
• Silver/gold schema contracts tested in CI (dbt tests or equivalent)
• Catalog is the only access path — no direct bucket reads for consumers
• Row/column policies applied; PII columns classified
• Partition strategy matches dominant query filters; reviewed quarterly
• Commit failures, small-file counts, and scan bytes alerting configured
• Freshness SLOs published per layer and monitored
• Cross-region replication or backup for gold and compliance tables
• Catalog backed up; restore path tested
• Engine/format version compatibility matrix maintained before protocol upgrades

Related Guides

Prerequisites

• ETL — the pipelines that feed every layer
• Data Lakes — the storage substrate lakehouses build on
• Data Warehouses — the semantics lakehouses borrow

Core Concepts

• AI System Architecture — where the lakehouse sits under AI serving layers

AI Consumers

• Embeddings — pipelines fed from curated tables
• Vector Databases — serving layers that sync from gold tables
• Metadata Filtering — governed table attributes improving retrieval

Diagram: Learning path through the data engineering cluster

flowchart LR
    ETL[ETL] --> DL[Data Lakes]
    ETL --> DW[Data Warehouses]
    DL --> LH[Lakehouse]
    DW --> LH
    LH --> AISA[AI System Architecture]
    LH --> EMB[Embeddings]
    EMB --> VDB[Vector DBs]

Master the two systems a lakehouse unifies first; then follow the data into AI serving layers.

Interview Questions

What problem do open table formats solve that Parquet alone does not?

Parquet is a file format — columnar, compressed, statistics-rich — but a "table" of raw Parquet files is just a directory listing. Table formats add a transactional metadata layer: atomic commits, consistent snapshots, schema evolution, and time travel. They turn concurrent writes from a corruption risk into an optimistic-concurrency retry.

How does an atomic commit work on object storage that has no transactions?

Data files are written first and are invisible because nothing references them. The commit is a single atomic pointer swap — a compare-and-swap on the catalog's current-metadata reference (Iceberg) or creation of the next ordered log file (Delta). Readers resolving the table always see either the old snapshot or the new one, never a mix.

Delta vs Iceberg — how would you choose?

Platform gravity first: Databricks-centric stacks get the most from Delta (Unity Catalog, deletion vectors, UniForm for interop). Multi-engine platforms — Spark plus Trino plus Snowflake external tables — favor Iceberg for its engine-neutral spec, hidden partitioning, and partition evolution. High-churn CDC ingestion is Hudi's niche. The wrong answer is per-team preference producing mixed formats on one lake.

What is the small files problem and how do you fix it?

Frequent small commits (streaming micro-batches) produce thousands of tiny files per partition. Planning slows, scan parallelism collapses, and metadata bloats. Fix: scheduled compaction rewriting small files into 128–512 MB targets, plus manifest rewrites and snapshot expiry so metadata shrinks with the data.

Why shouldn't a user-facing service query the lakehouse directly?

Object-storage latency plus query planning gives tens of milliseconds to seconds per lookup — fine for analytics, wrong for request paths. Production systems sync serving views into low-latency stores (feature stores, key-value stores, vector databases) and keep the lakehouse as the governed source of truth.

How does time travel support ML reproducibility?

Every commit creates an immutable snapshot. Training jobs record the snapshot ID they read; any later audit or retrain queries FOR VERSION AS OF that ID and gets byte-identical inputs — no "the table changed since training" ambiguity.

What does the medallion architecture buy you over one big curated layer?

Separation of failure domains and policies. Bronze preserves raw replayable history with drift tolerance; silver applies contracts so consumers get stable schemas; gold carries business-facing SLAs. Bad data caught at silver never reaches dashboards, and any layer can be rebuilt from the one below it.

Key Takeaways

• A lakehouse is warehouse semantics (ACID, schema, time travel) on lake economics (open files on object storage) — the table format is the load-bearing component.
• Writes stage immutable files and commit via one atomic metadata swap; reads prune by statistics before touching data. Metadata hygiene is performance.
• Medallion layers (bronze/silver/gold) put raw, cleaned, and business-ready tables on one storage layer with per-layer policies instead of separate systems.
• Choose Delta for Databricks gravity, Iceberg for multi-engine neutrality, Hudi for CDC-heavy ingestion — and enforce one format per domain.
• Production success is compaction, snapshot retention, catalog governance, and contracts — not engine choice.
• Keep serving out of the lakehouse: feature stores and vector databases sync from it; the lakehouse stays the governed source of truth for analytics and AI alike.

FAQs

Is a lakehouse the same as a data lake?

No. A data lake is storage. A lakehouse adds table semantics — ACID commits, schema enforcement, time travel — on top of that storage via open table formats.

Do I still need a data warehouse?

Many teams run both: the lakehouse for open storage and heavy transforms, the warehouse engine for governed BI and familiar SQL UX — increasingly reading the same Iceberg tables externally, so the data exists once.

Delta Lake vs Iceberg — which should I pick?

Delta if you are all-in on Databricks/Spark; Iceberg if you need multiple query engines and long-term vendor neutrality. Both are production-proven; mixing them per team is the only clearly wrong answer.

What is time travel actually used for?

Debugging bad pipeline runs (query the table as of before the incident), instant rollback after bad writes, regulatory audits, and reproducible ML training via pinned snapshot IDs.

How does a lakehouse relate to AI systems?

Training data, feature tables, and document metadata live in governed lakehouse tables; embedding pipelines and retrieval systems consume them and sync serving copies into specialized stores. See AI System Architecture for the full picture.

What causes lakehouse query slowdowns?

Too many small files, missing or wrong partitioning, stale table statistics defeating file pruning, and bloated snapshot metadata. Compaction, statistics maintenance, and snapshot expiry fix the vast majority.

Can I run a lakehouse without Spark?

Yes. Trino reads and writes Iceberg; DuckDB reads Iceberg and Delta for local analysis; Flink handles streaming writes; warehouse engines read external tables. Spark remains the most common transform engine, not a requirement.

Is "lakehouse" just Databricks marketing?

The term came from Databricks, but the pattern is vendor-neutral and multi-vendor in practice: Netflix built Iceberg, Uber built Hudi, and Snowflake, AWS, and Google all ship Iceberg support. The architecture outgrew its coiner.

References

• Delta Lake documentation
• Apache Iceberg documentation
• Apache Hudi documentation
• Databricks: What is a Lakehouse?
• Lakehouse: A New Generation of Open Platforms (CIDR 2021)

Further Reading

• Apache Iceberg spec
• Delta Lake transaction log protocol
• Trino Iceberg connector
• dbt on the lakehouse
• DuckDB Iceberg extension