Data Lakes Guide

TL;DR

• A data lake is centralized object storage for raw data at any scale - structured, semi-structured, and unstructured files in open formats (Parquet, JSON, Avro).

• Schema-on-read means structure is applied at query time, not enforced at ingest - maximum flexibility, governance responsibility shifts to consumers.

• Partitioning and file format choices dominate query performance - more than cluster size for most workloads.

• Without governance, lakes become data swamps - cataloging, access control, and lifecycle policies are mandatory.

• Lakes pair with ETL/ELT pipelines that land raw data, then transform into curated layers or warehouse tables.

Why This Matters

Modern organizations generate data faster than any warehouse schema can anticipate. Application logs, clickstreams, IoT sensors, ML model outputs, PDF contracts, and API exports arrive in incompatible formats from dozens of systems.

Forcing everything into rigid warehouse tables before storage creates bottlenecks: schema debates delay ingestion, new sources wait months for pipeline development, and semi-structured JSON gets shredded into hundreds of sparse columns nobody maintains.

Data lakes solve this by storing everything now, structuring later. S3 or GCS becomes the system of record for raw data. Analysts, data scientists, and ETL jobs apply schema when they read - enabling exploration, ML feature engineering, and gradual curation without upfront modeling tax.

Every major cloud analytics platform - Databricks, Snowflake external tables, BigQuery federated queries - assumes a lake layer exists underneath.

The Problem Data Lakes Solve

Before lakes, organizations faced painful tradeoffs:

Data lakes do not replace data warehouses. They complement them - lakes hold raw and exploratory data; warehouses serve curated, query-optimized analytics.

What Is a Data Lake?

A data lake is a storage architecture - typically built on object storage (Amazon S3, Google Cloud Storage, Azure Data Lake Storage) - that holds data files in open formats without requiring upfront schema definition.

Core properties:

• Schema-on-read - Structure interpreted at query time (Spark SQL, Athena, DuckDB).

• Scale - Petabytes on commodity storage; compute scales independently.

• Format diversity - Parquet, ORC, JSON, CSV, Avro, images, audio, video.

• Immutability - Append new files; avoid in-place updates on raw layer.

• Decoupled compute - Query engines (Spark, Trino, Presto) read the same files.

RAG couples a dense vector index of external knowledge with a sequence-to-sequence generator. At query time, the retriever selects relevant passages and the generator conditions its answer on that evidence.

Schema-on-Read vs Schema-on-Write

Schema-on-read is not "no schema." Curated layers apply schema-on-write discipline. The lake pattern uses medallion architecture:

Tip

Never query bronze directly in production dashboards. Raw zones are for engineering and replay; gold zones feed business users.

File Formats and Partitioning

File Format Comparison

Default recommendation: Parquet with Snappy compression for analytics layers. JSON in bronze only.

Partitioning Strategy

Partition by columns used in WHERE clauses - typically date:

s3://datalake/events/
  year=2026/month=07/day=06/
    part-00000.parquet
    part-00001.parquet

Warning

Over-partitioning creates millions of tiny files. Target 128MB–1GB per file. Run compaction jobs on schedule.

Architecture

A production data lake has layers beyond storage:

A production RAG stack separates offline indexing (load, chunk, embed, store) from online querying (embed query, retrieve, rerank, generate).

Decouple storage from compute. The same Parquet files should be queryable by Spark today and Trino tomorrow without data movement.

Real Production Example

An e-commerce company lands clickstream events to S3, curates with Spark, and exposes gold tables to Snowflake.

# Daily bronze → silver curation with Delta Lake
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, to_date

spark = SparkSession.builder.appName("silver_events").getOrCreate()

# Read bronze JSON (schema-on-read)
bronze = spark.read.json("s3://datalake/bronze/events/2026/07/06/")

# Transform - apply schema, deduplicate
silver = (
    bronze
    .filter(col("event_id").isNotNull())
    .dropDuplicates(["event_id"])
    .select(
        col("event_id"),
        col("user_id"),
        col("event_type"),
        col("properties"),
        col("timestamp").cast("timestamp").alias("event_ts"),
        to_date("timestamp").alias("event_date"),
    )
)

# Write silver as partitioned Parquet via Delta (schema-on-write for this layer)
(
    silver.write
    .format("delta")
    .mode("append")
    .partitionBy("event_date")
    .save("s3://datalake/silver/events/")
)

# Register in catalog for Athena / Trino
spark.sql("""
    CREATE TABLE IF NOT EXISTS catalog.silver.events
    USING DELTA
    LOCATION 's3://datalake/silver/events/'
""")

Downstream Snowflake reads gold aggregates via external table or Snowpipe. Raw bronze preserved for 90 days for replay.

Design Decisions

⚠ Common Mistakes

1. Data swamp - no catalog. Files exist but nobody can find them. Invest in metadata catalog from day one.

2. Tiny files problem. Streaming ingest creates thousands of 1MB files. Schedule compaction.

3. Querying bronze in production. Raw data has duplicates, nulls, and schema inconsistencies. Curate first.

4. No lifecycle policies. Bronze grows forever; storage costs explode. S3 lifecycle rules to Glacier or delete.

5. Wrong partition key. Partitioning by user_id with millions of users creates unlistable prefixes.

6. Ignoring access control. S3 bucket policies without column/row-level security expose PII across teams.

7. No immutability discipline. Overwriting raw files breaks reproducibility and audit trails.

Where It Breaks Down

Low-latency analytics. Lakes optimize throughput, not sub-second queries. Serve dashboards from warehouse or lakehouse gold tables with pre-aggregation.

Transactional workloads. Lakes are not OLTP replacements. Do not use S3 as your primary application database.

Governance at scale without investment. Schema-on-read shifts quality burden to consumers. Without data contracts and validation on silver/gold, trust collapses.

Concurrent writers on plain Parquet. Race conditions corrupt data. Use Delta/Iceberg/Hudi for concurrent MERGE operations.

Production Checklist

Important

A data lake without a catalog and curated gold layer is a data swamp. Budget governance alongside storage.

Ecosystem

• Storage: Amazon S3, Google Cloud Storage, Azure Data Lake Storage Gen2.

• Table formats: Delta Lake, Apache Iceberg, Apache Hudi.

• Compute: Databricks, EMR, Spark, Trino, Amazon Athena, DuckDB.

• Catalog: AWS Glue, Unity Catalog, Apache Polaris, Hive Metastore.

• Ingestion: Kafka → S3, Airbyte, Fivetran, ETL pipelines.

• Governance: AWS Lake Formation, Apache Ranger, Immuta.

Related Technologies

• ETL: Pipelines that land and transform lake data.

• Data Warehouses: Curated analytics layer often fed from lake gold zone.

• Lakehouse: Architecture unifying lake storage with warehouse semantics.

Learning Path

Prerequisites: ETL

Next topics: Data Warehouses · Lakehouse

Estimated time: 45 min · Difficulty: Intermediate

FAQs

What is the difference between a data lake and a data warehouse?

A lake stores raw files with schema-on-read on cheap object storage. A warehouse stores structured, optimized tables with schema-on-write for fast SQL analytics. They are complementary.

What is schema-on-read?

Structure is applied when data is queried, not when it is stored. Allows ingesting diverse formats without upfront modeling. Quality enforcement happens in curated layers.

What is a data swamp?

A data lake without governance - no catalog, no quality checks, no ownership. Data goes in but nobody trusts or finds it.

When should I use Parquet vs JSON?

JSON for raw ingest and human readability. Parquet for any analytics layer - columnar compression and predicate pushdown make queries 10–100x faster.

What are Delta Lake, Iceberg, and Hudi?

Table formats on top of Parquet that add ACID transactions, schema evolution, time travel, and MERGE support. Essential for production pipelines with concurrent writers.

How do I avoid the small files problem?

Batch micro-batches before writing, use streaming compaction (Delta auto-optimize), or schedule daily compaction jobs that coalesce files per partition.

Can I run a data lake on-premise?

Yes - HDFS was the original lake storage. Modern on-prem deployments use MinIO (S3-compatible) with Spark and Iceberg.

How does a data lake support AI and ML?

Raw text, images, and events stay accessible for feature engineering, model training, and RAG document ingestion without warehouse schema constraints.

What partition strategy should I start with?

year/month/day on event timestamp. Measure query patterns for 30 days before adding dimensions.

Lake vs lakehouse - what is the difference?

A lakehouse adds warehouse-like features (ACID, SQL performance, governance) on lake storage via table formats. See Lakehouse for details.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• Data lakes store raw data cheaply at scale with schema-on-read flexibility.
• Medallion architecture (bronze/silver/gold) balances flexibility with governance.
• Parquet + date partitioning is the default analytics pattern; compact files regularly.
• Table formats (Delta, Iceberg) add ACID and MERGE for production pipelines.
• Lakes complement warehouses - raw in the lake, curated metrics in the warehouse.
• Without catalog, lifecycle policies, and curated layers, lakes become swamps.