Lakehouse engineering

Delta Lake

Transactional data lakes, performance and scale.

Patterns for MERGE, schema evolution, partitioning, compaction, optimization and reliable lakehouse workloads.

Start here

Curated path

MERGE & data changes

Partitioning & optimization

Reliability & scale

Cheat sheets

Browse all →
  • PySparkData TransformationIntermediate

    GroupBy and Aggregations

    Aggregate PySpark DataFrames safely by defining output grain, handling null grouping keys, choosing correct aggregate functions, validating totals, and controlling shuffle-heavy groupBy workloads.

  • PySparkData TransformationIntermediate

    Join DataFrames in PySpark

    Join PySpark DataFrames safely by validating key cardinality, choosing the right join type and strategy, avoiding row multiplication, and handling skew, nulls, and ambiguous columns.

  • PySparkPerformanceIntermediate

    PySpark Performance Quick Reference

    Troubleshoot and optimize PySpark workloads by separating scan, shuffle, skew, partitioning, caching, joins, driver pressure, and file-layout problems before changing cluster size.

  • PySparkData EngineeringIntermediate

    Read and Write Delta Tables

    Read and write Delta tables with PySpark using safe append, overwrite, merge, schema-evolution, partitioning, and maintenance patterns for production data pipelines.

  • PySparkData TransformationIntermediate

    Select and Filter DataFrames

    Select, project, filter, and validate PySpark DataFrames efficiently while avoiding ambiguous columns, null-logic mistakes, unnecessary scans, and driver-side filtering.

  • PySparkData TransformationIntermediate

    Window Functions

    Use PySpark window functions safely for ranking, latest-row selection, running totals, lag/lead analysis, and partitioned calculations without collapsing DataFrame grain.

Latest content

Browse all →