Data Lakes Guide

TL;DR

• A data lake is centralized object storage for raw data at any scale - structured, semi-structured, and unstructured files in open formats (Parquet, JSON, Avro) on S3, GCS, or ADLS.

• Schema-on-read means structure is applied at query time, not enforced at ingest - maximum ingest flexibility, but governance responsibility shifts to consumers and curated layers.

• Partitioning and file format choices dominate query performance - more than cluster size for most workloads. Default to Parquet with date partitioning and 128MB–1GB files.

• Medallion architecture (bronze/silver/gold) is the standard organization pattern - raw fidelity in bronze, cleaned and typed data in silver, governed business aggregates in gold.

• Without governance, lakes become data swamps - cataloging, access control, compaction, and lifecycle policies are mandatory, not optional extras.

• Lakes pair with ETL/ELT pipelines that land raw data, then transform into curated layers or warehouse tables. Table formats like Iceberg and Delta Lake add ACID guarantees on top.

Quick Decision Guide

Who this guide is for

• Data engineers designing storage layers, ingestion pipelines, and medallion promotion logic.
• ML/AI engineers who need raw features, training corpora, and document stores for AI systems.
• Architects and technical leads deciding between lake, warehouse, and lakehouse patterns.
• Analysts and analytics engineers who query lake tables through Trino, Athena, or DuckDB and need to understand why layout matters.

Learning Path

Prerequisites: ETL - understand how data moves before studying where it lands.

This guide: storage architecture, schema-on-read, medallion layers, partitioning, table formats, production operations.

Next: Data Warehouses for the curated analytics layer, then Lakehouse for the converged architecture.

On this page

• Why This Matters
• The Problem Data Lakes Solve
• How We Got Here
• What Is a Data Lake?
• How Data Lakes Work
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Use a Data Lake
• Running in Production
• Production Checklist
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References

Why This Matters

Modern organizations generate data faster than any warehouse schema can anticipate. Application logs, clickstreams, IoT sensor readings, ML model outputs, PDF contracts, and API exports arrive in incompatible formats from dozens of systems - often before anyone knows exactly how they will be used.

Forcing everything into rigid warehouse tables before storage creates bottlenecks: schema debates delay ingestion, new sources wait months for pipeline development, and semi-structured JSON gets shredded into hundreds of sparse columns nobody maintains. Meanwhile, the data your ML team needed - the full raw payload, the original document, the unaggregated event stream - was discarded at load time.

Data lakes solve this by storing everything now, structuring later. S3 or GCS becomes the system of record for raw data. Analysts, data scientists, and ETL jobs apply schema when they read - enabling exploration, ML feature engineering, and gradual curation without an upfront modeling tax.

Engineering Insight

Every major cloud analytics platform assumes a lake layer exists underneath. Databricks is built directly on lake storage; Snowflake reads it through external tables and Iceberg catalogs; BigQuery federates over GCS. If you work in data or AI engineering, you will operate on top of a data lake whether you designed it or inherited it - and the difference between a usable lake and a swamp is a handful of decisions covered in this guide.

The economics reinforce the architecture. Object storage costs roughly $0.023/GB/month on S3 Standard, with cheaper tiers below that - one to two orders of magnitude cheaper than warehouse-managed storage was when the pattern emerged. Storing everything and deciding later became rational, and the discipline shifted from "what do we keep?" to "how do we organize what we kept?"

The Problem Data Lakes Solve

Before lakes, organizations faced painful tradeoffs:

Two of these deserve emphasis because they drive most real adoptions:

Reprocessing. Business logic changes. A sessionization rule gets fixed, a currency conversion was wrong, a new attribution model launches. If you only kept transformed warehouse tables, history is gone. An immutable bronze layer means any downstream table can be rebuilt from source at any time - the lake is the replay log for your entire analytics estate.

AI and ML workloads. Feature engineering wants raw events, not pre-aggregated metrics. Training corpora include text, images, and audio that never fit warehouse tables. Document ingestion for AI systems reads PDFs and HTML straight from lake storage. The lake is where AI workloads and analytics workloads share a single copy of the data.

Data lakes do not replace data warehouses. They complement them - lakes hold raw and exploratory data; warehouses serve curated, query-optimized analytics. The lakehouse pattern narrows the gap, but the division of responsibility remains the right mental model.

How We Got Here

The data lake is the third generation of "store everything" architecture, and each generation fixed the previous one's operational pain.

Diagram: Evolution of data lake storage

timeline
    title From HDFS to open table formats
    2006-2012 : Hadoop and HDFS : MapReduce on commodity clusters : Storage and compute coupled
    2013-2016 : Cloud object storage era : S3 and GCS replace HDFS : Spark replaces MapReduce : Schema-on-read goes mainstream
    2017-2020 : Table format emergence : Delta Lake, Apache Iceberg, Hudi : ACID transactions on Parquet : Time travel and MERGE
    2021-present : Lakehouse convergence : Unity Catalog, Polaris, Glue : Warehouses read Iceberg natively : Lake as shared foundation for BI and AI

Each generation kept the core idea - cheap raw storage - while fixing coupling, consistency, and governance.

Hadoop era (2006–2012). Google's GFS and MapReduce papers inspired Hadoop: HDFS spread files across commodity servers, MapReduce processed them in place. The term "data lake" was coined by James Dixon (Pentaho, 2010) to contrast with the data mart's bottled, pre-packaged water. The pain: clusters coupled storage and compute, so scaling one meant paying for both, and operating HDFS required dedicated teams.

Cloud object storage era (2013–2016). S3 offered eleven nines of durability, effectively infinite scale, and no cluster to babysit. Spark replaced MapReduce with in-memory DAG execution, and engines like Presto (now Trino) queried files directly with SQL. Storage and compute decoupled - the defining economic property of the modern lake. The pain: plain files on S3 have no transactions. Concurrent writers corrupted datasets, failed jobs left partial files, and "eventually consistent listings" caused missing-data bugs.

Table format era (2017–2020). Databricks released Delta Lake, Netflix built Apache Iceberg, and Uber built Apache Hudi - all solving the same problem: a metadata layer over Parquet files that provides ACID transactions, schema evolution, time travel, and safe concurrent writes. Suddenly a lake could support MERGE, DELETE (critical for GDPR), and streaming upserts.

Lakehouse convergence (2021–present). Catalogs (Unity Catalog, AWS Glue, Apache Polaris) added governance, lineage, and fine-grained access control. Warehouses gained the ability to read Iceberg and Delta tables natively, and engines from DuckDB to Spark to Snowflake now query the same physical files. The lake became the shared storage foundation of the lakehouse.

The lesson from this history: every failure of the lake pattern was a governance or consistency failure, not a storage failure - and each generation's fix (decoupling, table formats, catalogs) addressed exactly that.

What Is a Data Lake?

A data lake is a storage architecture - typically built on object storage (Amazon S3, Google Cloud Storage, Azure Data Lake Storage) - that holds data files in open formats without requiring upfront schema definition.

Core properties:

• Schema-on-read - Structure is interpreted at query time by the engine (Spark SQL, Athena, DuckDB, Trino), not enforced when files land.

• Scale - Petabytes on commodity object storage; compute scales independently and can be turned off entirely when idle.

• Format diversity - Parquet, ORC, JSON, CSV, Avro, plus unstructured assets: images, audio, video, PDFs.

• Immutability - Append new files; avoid in-place updates on the raw layer. Immutability is what makes replay, audit, and reproducible ML possible.

• Decoupled compute - Multiple query engines read the same files concurrently without data movement. The same Parquet dataset can serve a Spark training job, a Trino dashboard query, and a DuckDB notebook simultaneously.

• Open formats - No proprietary storage layer. Data outlives any single vendor or engine choice.

What a data lake is not: it is not a database, not a warehouse, and not a place to dump files without a plan. A lake is a contract - "raw data lives here, in these formats, organized this way, discoverable through this catalog" - and the storage service is just the substrate for that contract.

How Data Lakes Work

Three mechanisms determine whether a lake performs well or becomes unusable: schema-on-read, file formats, and partitioning. They interact - format and layout decisions made at write time determine what schema-on-read costs at query time.

Schema-on-Read vs Schema-on-Write

Schema-on-read is not "no schema." It is deferred schema: the reader supplies the interpretation. In practice, mature lakes apply schema-on-write discipline to curated layers - bronze stays permissive, silver and gold enforce types, constraints, and contracts. The flexibility is a property of the raw zone, not the whole lake.

Diagram: How a schema-on-read query executes

sequenceDiagram
    participant U as Analyst / Job
    participant E as Query engine (Trino/Spark)
    participant C as Catalog (Glue/Unity)
    participant S as Object storage (S3)
    U->>E: SELECT ... WHERE event_date = '2026-07-20'
    E->>C: Resolve table → schema, format, partitions
    C-->>E: Metadata + partition locations
    E->>S: Read only matching partition files
    S-->>E: Parquet column chunks (projected columns only)
    E-->>U: Results (schema applied at read)

The catalog turns "files in a bucket" into a table; partition pruning and columnar projection determine how few bytes the engine must scan.

File Formats

Columnar formats matter because analytics queries touch few columns of many rows. Parquet stores each column's values contiguously with per-chunk statistics (min/max, null counts), so engines skip entire row groups without reading them (predicate pushdown) and read only projected columns. Typical result: 10–100x less I/O than JSON for the same query.

Default recommendation: Parquet with Snappy compression (ZSTD when storage cost outweighs CPU) for silver and gold. JSON or Avro in bronze only, converted on promotion.

Partitioning

Partitioning encodes a filter column into the directory layout so engines can skip irrelevant files entirely:

s3://datalake/silver/events/
  event_date=2026-07-19/
    part-00000.parquet
  event_date=2026-07-20/
    part-00000.parquet
    part-00001.parquet

A query with WHERE event_date = '2026-07-20' lists and reads only that prefix. Partition by columns that dominate your WHERE clauses - almost always date first:

Two failure modes bracket every partitioning decision:

• Over-partitioning creates millions of tiny files. Each file has fixed open/list overhead, so a query that should scan 10 large files instead scans 100,000 small ones and spends its time on metadata, not data.
• Under-partitioning forces full scans. A 2TB unpartitioned table means every query pays for 2TB of I/O regardless of its filter.

Warning

Target 128MB–1GB per file. Streaming ingest naturally produces small files - schedule compaction (Delta OPTIMIZE, Iceberg rewrite_data_files) or the lake degrades within weeks. Modern table formats also offer hidden partitioning (Iceberg transforms like days(ts)) and clustering (Delta liquid clustering) that reduce the cost of getting this wrong.

Architecture

A production data lake is a stack, not a bucket. Storage is the cheapest and least interesting layer:

The defining architectural principle: decouple storage from compute. The same Parquet files should be queryable by Spark today and Trino tomorrow without data movement. Any component in the table above should be replaceable without rewriting data.

Medallion Architecture

The standard organization pattern layers the lake into zones of increasing quality:

Each promotion step adds guarantees and removes flexibility. Bronze answers "what did the source actually send?"; silver answers "what happened, cleanly?"; gold answers "what does the business need to know?". Data contracts live at the silver boundary: ingestion can land anything, but nothing enters silver without passing type checks, deduplication, and validation.

Tip

Never query bronze directly from production dashboards. Raw zones are for engineering and replay; gold zones feed business users. If an analyst is writing JSON-parsing SQL against bronze, that is a signal a silver table is missing.

Step-by-Step Flow

A record's journey from source system to dashboard follows the same path in nearly every production lake:

Diagram: Ingest-to-gold data flow

flowchart TD
    A[Sources: apps, APIs, CDC, IoT] --> B{Ingestion}
    B -->|Streaming| C[Kafka / Kinesis → micro-batches]
    B -->|Batch| D[ETL / Airbyte / Fivetran]
    C --> E[Bronze: raw JSON/Avro, append-only, partitioned by ingest date]
    D --> E
    E --> F[Silver job: validate, cast types, dedupe, conform]
    F -->|Rejects| Q[Quarantine + alert]
    F --> G[Silver: Delta/Iceberg Parquet, partitioned by event date]
    G --> H[Gold job: joins, aggregates, business logic]
    H --> I[Gold: governed marts and metrics]
    I --> J[BI dashboards / Trino / Athena]
    I --> K[Warehouse external tables / sync]
    G --> L[ML feature pipelines and training]

Every arrow is a job with an owner, a schedule, and a data-quality gate; the diagram is also your lineage map.

Walking through the stages:

1. Ingest. Streaming sources (Kafka, Kinesis) land micro-batches every few minutes; batch sources land on schedule via ETL tools. Ingestion writes to bronze only - no transformation beyond adding ingest metadata (_ingested_at, _source, _batch_id).

2. Bronze landing. Files are append-only and partitioned by ingest date. Nothing is ever modified or deleted here inside the retention window - this layer is the replay log.

3. Silver promotion. A scheduled job reads new bronze partitions, applies schema (casting, renaming, flattening), deduplicates on business keys, validates rows, and writes to a Delta or Iceberg table partitioned by event date (not ingest date). Rows failing validation go to a quarantine table with an alert - never silently dropped.

4. Gold promotion. Business logic - joins across silver tables, sessionization, aggregates, slowly-changing dimensions - produces marts. Increasingly this layer is expressed in dbt running against Trino or Spark SQL, giving version-controlled, tested transformations.

5. Serve. Gold tables feed BI directly via Trino/Athena, sync to a warehouse for high-concurrency dashboards, and feed reverse-ETL. Silver feeds ML feature pipelines, which prefer cleaned-but-unaggregated data.

Latency through this flow ranges from minutes (streaming + Delta) to a day (nightly batch). Choose per table based on consumer needs - most gold tables genuinely need only daily freshness.

Real Production Example

An e-commerce company lands clickstream events to S3 via Kinesis Firehose, curates with Spark on Databricks, and exposes gold tables to analysts through Trino and Snowflake external tables.

# Daily bronze → silver curation with Delta Lake
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, to_date, current_timestamp, input_file_name

spark = SparkSession.builder.appName("silver_events").getOrCreate()

BRONZE = "s3://datalake/bronze/events/ingest_date=2026-07-21/"
SILVER = "s3://datalake/silver/events/"
QUARANTINE = "s3://datalake/quarantine/events/"

# 1. Read bronze JSON (schema-on-read; permissive to capture bad rows)
bronze = (
    spark.read
    .option("mode", "PERMISSIVE")
    .option("columnNameOfCorruptRecord", "_corrupt")
    .json(BRONZE)
    .withColumn("_source_file", input_file_name())
)

# 2. Split valid rows from rejects - never silently drop data
valid = bronze.filter(col("_corrupt").isNull() & col("event_id").isNotNull())
rejects = bronze.subtract(valid)

if not rejects.isEmpty():
    rejects.withColumn("_quarantined_at", current_timestamp()) \
        .write.mode("append").format("delta").save(QUARANTINE)
    # Alerting hook: page if reject rate > 1% of batch

# 3. Apply schema and deduplicate (schema-on-write begins at silver)
silver = (
    valid
    .dropDuplicates(["event_id"])
    .select(
        col("event_id"),
        col("user_id"),
        col("event_type"),
        col("properties"),
        col("timestamp").cast("timestamp").alias("event_ts"),
        to_date("timestamp").alias("event_date"),
        current_timestamp().alias("_processed_at"),
    )
)

# 4. Idempotent MERGE - safe to re-run the job for the same day
from delta.tables import DeltaTable

if DeltaTable.isDeltaTable(spark, SILVER):
    (
        DeltaTable.forPath(spark, SILVER).alias("t")
        .merge(silver.alias("s"), "t.event_id = s.event_id")
        .whenNotMatchedInsertAll()
        .execute()
    )
else:
    silver.write.format("delta").partitionBy("event_date").save(SILVER)

# 5. Register in catalog so Trino / Athena / DuckDB can query it
spark.sql(f"""
    CREATE TABLE IF NOT EXISTS catalog.silver.events
    USING DELTA LOCATION '{SILVER}'
""")

# 6. Compact small files from streaming ingest
spark.sql("OPTIMIZE catalog.silver.events WHERE event_date >= current_date() - INTERVAL 7 DAYS")

Operational details that make this production-grade rather than a demo:

• Idempotency. The MERGE on event_id means re-running a failed or backfilled day cannot create duplicates. Every promotion job in the lake should be safely re-runnable.
• Quarantine, not drop. Corrupt and null-key rows are preserved with provenance (_source_file) so schema drift in the source is debuggable, and reject-rate alerts catch upstream breakage within one batch.
• Compaction inline. Streaming ingest produces small files; OPTIMIZE keeps recent partitions healthy without a separate pipeline.
• Catalog registration. The table is immediately queryable by Trino, Athena, and DuckDB - no copies, one physical dataset.

Downstream, a nightly dbt run builds gold aggregates (daily revenue by category, funnel conversion) from silver. Bronze is retained 90 days on S3 Standard-IA for replay, then expires via lifecycle policy.

Design Decisions

Common patterns worth naming:

• Write-audit-publish. Promotion jobs write to a staging location, run quality checks, then atomically swap into the production table (native in Iceberg via branch/commit). Consumers never see partially-written data.
• Kappa-style replay. Because bronze is immutable, fixing a transformation bug means redeploying the silver job and replaying affected partitions - no source re-extraction.
• Single physical copy, many engines. Resist making per-team copies. The catalog plus open formats exist precisely so Spark, Trino, and DuckDB share one dataset.

Comparisons

Lake vs Warehouse vs Lakehouse

Parquet vs JSON

Delta Lake vs Iceberg vs Hudi

All three deliver the core guarantees - ACID, time travel, schema evolution, MERGE. The decision is ecosystem alignment, not features:

Decision tree: choosing your lake storage strategy

flowchart TD
    A{Need updates, deletes,<br/>or concurrent writers?} -->|No, append-only archive| B[Plain Parquet<br/>+ date partitions]
    A -->|Yes| C{Primary platform?}
    C -->|Databricks| D[Delta Lake]
    C -->|Multi-engine /<br/>Snowflake + Trino + Spark| E[Apache Iceberg]
    C -->|Streaming CDC-first| F[Apache Hudi]
    B --> G{Dashboards need<br/>sub-second + high concurrency?}
    D --> G
    E --> G
    F --> G
    G -->|Yes| H[Sync gold to warehouse]
    G -->|No| I[Serve via Trino / Athena / DuckDB]

Table format choice is an ecosystem decision; serving choice is a latency-and-concurrency decision. Keep them independent.

Common Mistakes

1. Data swamp - no catalog. Files exist but nobody can find them, trust them, or knows who owns them. Register every production table in a catalog with schema and owner from day one; a lake without a catalog is a bucket.

2. Tiny files problem. Streaming ingest creates thousands of 1MB files; queries spend their time on file-open overhead. Schedule compaction (OPTIMIZE, rewrite_data_files) and monitor average file size per partition.

3. Querying bronze in production. Raw data has duplicates, nulls, and schema drift. Dashboards on bronze break silently and erode trust. Curate to silver first.

4. No lifecycle policies. Bronze grows forever; storage bills compound quietly. Set S3 lifecycle rules per zone - transition to infrequent access, then Glacier or delete.

5. Wrong partition key. Partitioning by user_id with millions of users creates unlistable prefixes and universal skew. Partition by date; add dimensions only with query-pattern evidence.

6. Ignoring access control. Bucket-level IAM without column/row-level security exposes PII to every team with lake access. Use Lake Formation, Unity Catalog, or Ranger for fine-grained grants.

7. No immutability discipline. Overwriting raw files in place breaks replay, audit trails, and reproducible ML. Bronze is append-only; corrections happen in silver.

8. Plain Parquet with concurrent writers. Two jobs writing the same prefix without a table format produce silent corruption or lost data. Any table with more than one writer needs Delta/Iceberg/Hudi.

9. Skipping data contracts at the silver boundary. If anything can be promoted to silver, silver is just bronze with extra steps. Enforce types, keys, and validation on promotion, with quarantine for failures.

Where It Breaks Down

Low-latency analytics. Lakes optimize throughput, not sub-second response. Object storage listing, file opening, and cold engine start put a floor on latency that no tuning removes. Serve high-concurrency dashboards from a warehouse or heavily-optimized gold tables with caching.

Small-data overhead. Below a few hundred GB, the lake stack (catalog, table format, Spark cluster, compaction jobs) costs more in operations than it saves in storage. Postgres or DuckDB over a handful of Parquet files is simpler and faster.

Governance at scale without investment. Schema-on-read shifts the quality burden to consumers. Without contracts, validation, ownership, and lineage on silver/gold, every consumer re-implements cleaning logic differently and trust collapses - the swamp failure mode is organizational, not technical.

Concurrent writers on plain Parquet. Race conditions between writers corrupt datasets undetectably. This is the single most common cause of "the lake lost data" incidents; table formats exist because of it.

Metadata at extreme scale. Millions of partitions or billions of files stress catalogs and planning phases. Iceberg's manifest design mitigates this, but tables need periodic metadata maintenance (snapshot expiry, manifest rewrite) or planning time degrades.

When NOT to Use a Data Lake

• Transactional applications. Lakes are not OLTP. Point lookups, row-level updates at request latency, and application state belong in Postgres/MySQL/DynamoDB - never S3.

• Single-source, structured-only analytics. If all your data is already relational and fits a warehouse affordably, a lake adds architecture without adding value. Load it directly.

• Small teams, small data. Under ~100GB with a two-person data team, DuckDB + Parquet on a disk (or just the warehouse) delivers the benefits without the medallion machinery.

• Sub-second serving layers. Product features needing millisecond reads (recommendations, feature serving) require a dedicated online store (Redis, feature store), with the lake as the offline source.

• When you cannot fund governance. A lake without catalog, ownership, and lifecycle budget will become a swamp within a year. If the organization will only fund storage, use a managed warehouse where governance is built in.

Running in Production

Operating a lake is a set of continuous processes, not a one-time build:

Monitoring. Track per-table: ingest volume and lag, partition freshness (latest event_date vs now), reject/quarantine rates, average file size, and query scan bytes. Freshness alerts catch dead pipelines; scan-byte anomalies catch partitioning regressions.

Compaction and maintenance. Schedule OPTIMIZE/rewrite_data_files for hot partitions, expire old table-format snapshots (they hold storage), and vacuum orphaned files from failed writes. Budget maintenance compute at roughly 10–20% of transformation compute.

Cost management. Storage: lifecycle policies per zone, compression audits, snapshot expiry. Compute: partition pruning verification (a missing pruning predicate can 100x a query's cost on pay-per-scan engines like Athena), spot instances for batch, right-sized clusters. Tag jobs to teams for allocation.

Security. Encrypt at rest (SSE-KMS) and in transit; block public bucket access at the account level; fine-grained access via Lake Formation or Unity Catalog; column masking for PII; audit access logs. GDPR deletes require a table format - plan the erasure pipeline before the first PII lands.

Disaster recovery. Object storage is durable but not deletion-proof. Enable versioning on gold, cross-region replication for critical zones, and test a bronze→gold rebuild quarterly - the replay path is your real backup.

Backfills and schema evolution. Every promotion job idempotent and parameterized by date range. Schema changes flow: additive columns are safe (table formats handle evolution); breaking changes get a new table version with a migration window.

Important

A data lake without a catalog and a curated gold layer is a data swamp. Budget governance - catalog, ownership, contracts, lifecycle - alongside storage from day one, not as a phase two.

Production Checklist

• Bronze/silver/gold zones defined with documented promotion criteria and owners per table
• Parquet (Snappy/ZSTD) for all analytics layers; JSON/Avro confined to bronze
• Date-based partitioning with 128MB–1GB target file size; layout reviewed against real query patterns
• Table format (Delta/Iceberg/Hudi) on every table with updates, deletes, or concurrent writers
• All production tables registered in a catalog with schema, description, and owner
• Compaction and snapshot-expiry jobs scheduled; average file size monitored
• Lifecycle policies per zone (bronze → IA → Glacier/delete; silver/gold per compliance)
• Fine-grained access control (column/row-level) and PII masking in place; public access blocked
• Data-quality gates with quarantine tables and reject-rate alerts at the silver boundary
• Freshness, volume, and scan-cost monitoring with paging alerts on dead pipelines
• All promotion jobs idempotent and replayable by date range; backfill runbook tested
• GDPR/erasure pipeline designed and tested before PII ingestion

Related Guides

Diagram: Data engineering learning path

flowchart LR
    A[ETL] --> B[Data Lakes]
    B --> C[Data Warehouses]
    B --> D[Lakehouse]
    C --> D
    D --> E[AI System Architecture]

Start with pipelines, then storage, then the converged lakehouse, then how AI systems consume it all.

• ETL - The pipelines that land raw data in bronze and promote it through silver and gold. Prerequisite for this guide.

• Data Warehouses - The curated, high-concurrency analytics layer typically fed from the lake's gold zone. Read next to understand the serving side.

• Lakehouse - The convergence: warehouse semantics (ACID, governance, performance) implemented directly on lake storage via table formats.

• AI System Architecture - How AI platforms consume lake data: training corpora, feature pipelines, and document ingestion for retrieval systems.

Interview Questions

Q: What is schema-on-read and what is its main tradeoff? Structure is applied at query time rather than enforced at ingest. Tradeoff: maximum ingest flexibility and source fidelity, but data quality is unvalidated at rest - the burden shifts to consumers unless curated layers (silver/gold) reintroduce schema-on-write discipline.

Q: Explain medallion architecture and why each layer exists. Bronze preserves raw source fidelity for replay and audit; silver applies types, deduplication, and validation for trustworthy engineering use; gold applies business logic for governed consumption. Each layer trades flexibility for guarantees, and the silver boundary is where data contracts are enforced.

Q: Why do table formats like Delta and Iceberg exist? What do plain Parquet files lack? Plain files on object storage have no transactions: concurrent writers corrupt data, failed jobs leave partial files, and there is no atomic schema change, MERGE, DELETE, or time travel. Table formats add a metadata layer over Parquet providing ACID commits, snapshot isolation, schema/partition evolution, and record-level operations (essential for GDPR).

Q: How would you fix a lake suffering from the small files problem? Immediate: run compaction (Delta OPTIMIZE, Iceberg rewrite_data_files) targeting 128MB–1GB files. Structural: batch micro-batches before writing, reduce partition granularity if daily volumes are small, enable auto-compaction on streaming writes, and add average-file-size monitoring so it cannot silently recur.

Q: A query on a 2TB partitioned table scans the full 2TB. What do you check? Whether the filter predicate matches the partition column exactly (functions on the partition column defeat pruning), whether the engine resolved partitions via the catalog, whether the table is actually partitioned as assumed, and whether statistics/manifests are stale. On Iceberg, check hidden-partition transforms align with the filter.

Q: When would you recommend a warehouse over a lake? When data is fully structured from few sources, consumers are high-concurrency BI dashboards needing sub-second latency, and the team cannot fund lake governance. The lake wins for raw diversity, ML workloads, replay, and PB-scale economics; most mature stacks run both, lake feeding warehouse.

Q: How do you handle GDPR right-to-erasure in a data lake? Requires a table format: run DELETE (Delta/Iceberg) which rewrites affected files transactionally, then vacuum/expire old snapshots so deleted data is physically removed within the compliance window. For bronze, either keep PII out (tokenize at ingest) or apply crypto-shredding with per-user keys.

Key Takeaways

• A data lake is cheap, open-format object storage plus a contract: zones, formats, partitioning, and a catalog.
• Schema-on-read is a property of the raw zone; production lakes reintroduce schema-on-write at silver and gold.
• Parquet + date partitioning + 128MB–1GB files is the default that makes everything downstream fast.
• Table formats (Delta, Iceberg, Hudi) are mandatory for any table with updates or concurrent writers.
• The medallion pattern turns "store everything" from a liability into a replayable, governed asset.
• Lakes complement warehouses; the lakehouse converges them. Governance funding, not technology, decides swamp vs asset.

FAQs

What is the difference between a data lake and a data warehouse?

A lake stores raw files with schema-on-read on cheap object storage in open formats. A warehouse stores structured, optimized tables with schema-on-write for fast, high-concurrency SQL analytics. They are complementary: raw and exploratory data in the lake, governed metrics in the warehouse.

What is schema-on-read?

Structure is applied when data is queried, not when it is stored. This allows ingesting diverse formats without upfront modeling; quality enforcement happens later, in curated silver/gold layers.

What is a data swamp?

A data lake without governance - no catalog, no quality checks, no ownership. Data goes in but nobody can find it or trust it. Prevented by cataloging, data contracts at the silver boundary, and lifecycle policies.

When should I use Parquet vs JSON?

JSON for raw ingest and debugging readability, in bronze only. Parquet for any analytics layer - columnar layout, compression, and predicate pushdown make queries 10–100x faster and files 5–10x smaller.

What are Delta Lake, Iceberg, and Hudi?

Table formats layered on Parquet that add ACID transactions, schema evolution, time travel, and MERGE/DELETE support. Choose by ecosystem: Delta for Databricks-centric stacks, Iceberg for engine neutrality, Hudi for streaming CDC.

How do I avoid the small files problem?

Batch micro-batches before writing, enable auto-compaction on streaming writes, and schedule compaction jobs (Delta OPTIMIZE, Iceberg rewrite_data_files) that coalesce files to 128MB–1GB per partition.

Can I run a data lake on-premise?

Yes - HDFS was the original lake storage. Modern on-prem deployments use MinIO (S3-compatible object storage) with Spark or Trino and Iceberg, keeping the same open-format architecture.

How does a data lake support AI and ML?

Raw text, images, and full event payloads stay accessible for feature engineering, model training, and document ingestion - data a warehouse would have discarded or flattened. See AI System Architecture for how AI platforms consume lake data.

What partition strategy should I start with?

Partition by event date (event_date=YYYY-MM-DD or Iceberg days(ts)). Measure real query patterns for 30 days before adding any second dimension, and verify file sizes stay above 128MB.

Lake vs lakehouse - what is the difference?

A lakehouse adds warehouse-like capabilities - ACID transactions, SQL performance, unified governance - directly on lake storage via table formats and catalogs. The lake is the storage foundation; the lakehouse is the full platform. See Lakehouse.

Do I need Spark to run a data lake?

No. Spark dominates heavy transformation, but Trino/Athena cover SQL analytics, DuckDB handles single-node work on Parquet remarkably well, and dbt over a SQL engine can own the gold layer entirely.

References

• Amazon S3 Documentation - object storage foundations, lifecycle policies, and security controls
• Delta Lake Documentation - transaction log, OPTIMIZE, MERGE, and time travel
• Apache Iceberg Documentation - table spec, hidden partitioning, and snapshot management
• Apache Parquet Documentation - columnar format internals and encoding
• Apache Hudi Documentation - streaming upserts and record-level indexing
• AWS Lake Formation Documentation - fine-grained lake access control

Further Reading

• Databricks - The Big Book of Data Engineering - medallion patterns and Delta operations at scale
• Trino Documentation - federated SQL over lake tables
• DuckDB Documentation - local analytics directly on Parquet and Iceberg
• dbt Documentation - version-controlled transformations for silver and gold layers
• Dremel paper (Google, 2010) - the columnar-execution ideas behind Parquet