Delta Lake
Working with Billions of Rows in Delta Lake
What changes when Delta tables reach billions of rows: partitioning, file sizing, MERGE, incremental processing, small files, compaction and concurrent writers.
- Status
- Proposed
- Type
- Talk
- Formats
- Webinar · Internal session
- Level
- Advanced
- Duration
- 45–60 minutes
This is a proposed session topic. No public event is currently scheduled.
Who it is for
- Data engineers operating large Delta tables in Fabric, Databricks or Spark
- Engineers whose MERGE and maintenance jobs are getting slower every month
What attendees will learn
- How to choose a partitioning strategy, including when not to partition
- Why file size matters for reads, writes and the transaction log
- How MERGE works internally and how to keep it narrow
- How to process only what changed at this scale
- How small files appear, and how to plan compaction and maintenance
- What happens when several jobs write to the same table
Session overview
Design choices that are harmless at a million rows become expensive at a billion. A partition column that once looked sensible creates millions of files. A MERGE that took minutes takes hours, and maintenance jobs collide with loads. This session explains the mechanics behind those symptoms and the design and operational patterns that keep very large Delta tables fast and predictable.
Outline
- How Delta stores data: Parquet files, the transaction log and checkpoints.
- Partitioning: cardinality, data per partition, and clustering as an alternative.
- File sizing and small files: where they come from and what they cost.
- MERGE at scale: narrowing the target, pruning, and deduplicating the source.
- Incremental processing: change detection and idempotent reruns.
- Compaction and maintenance: OPTIMIZE, VACUUM and scheduling around loads.
- Concurrency: optimistic concurrency, conflicts, and designing writers that do not collide.
Related articles
Delta LakeIntermediate
Delta Table Maintenance in Microsoft Fabric: OPTIMIZE, VACUUM, V-Order and Small Files
Why Fabric Delta tables slow down over time and which maintenance actually helps: diagnosing small files, OPTIMIZE and auto compaction, VACUUM and retention, and when V-Order is worth it.
Microsoft FabricIntermediate
Incremental Loads in Microsoft Fabric: Watermarks, CDC and MERGE Without Reprocessing Everything
Stop full reloads in Microsoft Fabric without losing correctness: bounded watermark windows, Change Tracking vs CDC, Delta MERGE, deletes, late data and safe reruns.
Delta LakeAdvanced
Delta Lake MERGE Performance Patterns for Large Tables
Why Delta MERGE gets slow on large tables and the patterns that keep it fast: clean sources, pruning predicates, fewer touched files, sane file sizes, safe concurrency and knowing when not to MERGE.
Delta LakeBeginner
Delta Tables in Microsoft Fabric for SQL Server Engineers
What a Delta table really is, how it differs from a SQL Server table, and what to understand about writes, keys, time travel, maintenance and the SQL analytics endpoint before using Fabric Lakehouse tables in production.