Python
PySpark DataFrame Unit Testing with pytest: A Practical Guide
Learn how to unit test PySpark DataFrame transformations with pytest using small deterministic datasets, schema assertions, fixtures, edge cases and reusable testing patterns, and where unit tests stop and integration tests begin.
En este artículo
This guide answers one question:
How do I prove that a PySpark transformation is correct without running an entire production pipeline?
The answer is a unit test: a few rows in, the transformation, the expected rows out, on a local Spark session, in seconds. Three principles shape everything below:
- Test transformation logic with the smallest DataFrame that proves the behaviour.
- A Spark job doesn’t need millions of rows to prove that its business logic is correct. It needs the right three rows: the duplicate, the null, the cancelled order.
- Unit tests isolate transformation behaviour. Storage, orchestration and infrastructure belong in integration tests.
This is the hands-on companion to Testing a Production Data Platform, which explains where unit tests sit among data-quality checks, integration tests and CI/CD gates. Nothing here is specific to Microsoft Fabric; a short section at the end maps the boundaries onto Fabric.
Unit, integration and end-to-end: draw the boundary first
Most PySpark testing pain comes from a test that tries to be all three kinds at once: it reads a table, transforms it, writes it, and then nobody can tell which part broke.
- Unit testThe logic, in isolation
- Input DataFrameA few rows, written in the test
- TransformationA Python function
- Seconds, local SparkExpected DataFrame
- Integration testSpark plus real storage
- Read Delta
- Transform
- Real tables, test dataWrite Delta
- End-to-end testThe whole pipeline
- Source
- Pipeline and notebook
- Slow, few, criticalLakehouse and Warehouse
| Kind | Example | What it proves | Where it runs |
|---|---|---|---|
| Unit | input DataFrame → transform_orders → expected DataFrame |
The business logic is right for the cases you designed | Developer machine, every pull request |
| Integration | read Delta → transform → write Delta, a MERGE run twice |
Spark, Delta and the storage behave as the code assumes | A test environment with real storage |
| End-to-end | source → Fabric pipeline → notebook → Lakehouse → Warehouse |
The pieces are connected, configured and ordered correctly | A test workspace, before release |
This guide is about the first row, and about keeping the other two out of it.
The example: orders
Every example uses the same small domain, so the tests read as a specification of one transformation.
| Column | Type | Meaning |
|---|---|---|
order_id |
long |
Business key; an order can have several events |
customer_id |
long |
Who placed it |
amount |
decimal(18,2) |
Order value; money stays decimal |
status |
string |
paid, PAID , cancelled… as sent by the source |
event_timestamp |
timestamp |
When this version of the order happened |
The business rules, written down before any test:
- Normalize status: trim and upper-case. Known values are
PAID,PENDING,CANCELLED,REFUNDED; null or anything else becomesUNKNOWN. Unknown rows are kept, so they stay visible. - Discard invalid rows: an order needs an
order_idand acustomer_idto be attributed. Nothing else is discarded. - Deduplicate: keep the latest event per
order_id. On a timestamp tie,statusascending decides, soCANCELLEDbeatsPAID. - Revenue: only
PAIDorders with a known, non-negative amount count. Aggregate per customer:revenueandpaid_orders.
Structure the code so it can be tested
Notebook code often looks like this:
df = spark.read.table("bronze.orders")
df = df.withColumn("status", F.upper(F.trim("status")))
df = df.where(F.col("customer_id").isNotNull())
# ... forty more lines ...
df.write.mode("overwrite").saveAsTable("gold.customer_revenue")
To test the logic you would need the bronze.orders table, write access to gold, and a cleanup step, and a failure could come from any of the three. Separate the parts instead:
def read_orders(spark):
return spark.read.table("bronze.orders")
def write_revenue(revenue):
revenue.write.mode("overwrite").saveAsTable("gold.customer_revenue")
# Orchestration: the notebook or job entry point
revenue = transform_orders(read_orders(spark))
write_revenue(revenue)
transform_orders takes a DataFrame and returns a DataFrame. It has no spark.read, no write, no paths and no table names, which is what makes it unit-testable.
Keep transformation functions as pure as practical.
Spark transformations aren’t pure in the strict functional sense: they build a lazy plan that runs on a cluster, and functions such as current_timestamp() change between runs. The useful version of the rule is narrower: a transformation function receives every input as an argument and does no I/O. That alone makes it possible to call it from a test with a hand-written DataFrame.
- Integration testread_orders()Real table or files
- Unit testtransform_orders()DataFrame in, DataFrame out
- Integration testwrite_revenue()Real table, real format
Project layout
A small layout is enough:
pyproject.toml
src/
transforms/
__init__.py
orders.py
tests/
conftest.py
test_orders.py
pytest needs to import transforms from src. The simplest way is pytest’s pythonpath setting:
# pyproject.toml
[tool.pytest.ini_options]
pythonpath = ["src"]
testpaths = ["tests"]
markers = ["integration: needs real storage (Delta, a Lakehouse); excluded from the unit-test run"]
addopts = "-m 'not integration'"
The alternative, when the transformations ship as a wheel (for example as a custom library in a Fabric environment), is to install the package in editable mode (pip install -e .) and drop pythonpath. Either works; pick the one that matches how the code is deployed. The integration marker and addopts keep integration tests out of the default run; they come back in a separate CI step (pytest markers).
Pin PySpark to the Spark version you deploy on. Testing against Spark 4.x and running on Spark 3.5 (or the reverse) lets version differences through. In Microsoft Fabric, Runtime 2.0 runs Spark 4.1 with Python 3.13, and Runtime 1.3 runs Spark 3.5.5 with Python 3.11 (Fabric runtimes).
The transformation under test
The core of the module, exactly as tested (three smaller helpers used later, add_order_date, to_eur and enrich_with_segment, live in the same file and are shown with their tests):
# src/transforms/orders.py
"""Order transformations: pure DataFrame-in, DataFrame-out functions with no I/O."""
from pyspark.sql import DataFrame, Window
from pyspark.sql import functions as F
KNOWN_STATUSES = ["PAID", "PENDING", "CANCELLED", "REFUNDED"]
def normalize_status(orders: DataFrame) -> DataFrame:
"""Trim and upper-case status. Null or unknown values become UNKNOWN (kept, not dropped)."""
cleaned = F.upper(F.trim(F.col("status")))
return orders.withColumn(
"status",
F.when(cleaned.isin(KNOWN_STATUSES), cleaned).otherwise(F.lit("UNKNOWN")),
)
def discard_invalid(orders: DataFrame) -> DataFrame:
"""An order needs an order_id and a customer_id to be attributed. Everything else is kept."""
return orders.where(F.col("order_id").isNotNull() & F.col("customer_id").isNotNull())
def deduplicate_orders(orders: DataFrame) -> DataFrame:
"""Keep the latest event per order_id.
Tie-break (same event_timestamp): status ascending, so CANCELLED wins over PAID.
The tie-break is a business decision; it is written down so the result is deterministic.
"""
latest_first = Window.partitionBy("order_id").orderBy(
F.col("event_timestamp").desc(), F.col("status").asc()
)
return (
orders.withColumn("_rank", F.row_number().over(latest_first))
.where(F.col("_rank") == 1)
.drop("_rank")
)
def revenue_by_customer(orders: DataFrame) -> DataFrame:
"""Revenue counts PAID orders only, with a known, non-negative amount."""
return (
orders.where((F.col("status") == "PAID") & (F.col("amount") >= 0))
.groupBy("customer_id")
.agg(
F.sum("amount").alias("revenue"),
F.count("*").alias("paid_orders"),
)
)
def transform_orders(orders: DataFrame) -> DataFrame:
"""The full transformation: normalize, validate, deduplicate, aggregate."""
return revenue_by_customer(deduplicate_orders(discard_invalid(normalize_status(orders))))
Small functions composed into transform_orders give you two levels of test for free: each rule on its own, and the whole transformation end to end on a handful of rows.
The SparkSession fixture
# tests/conftest.py
import pytest
from pyspark.sql import SparkSession
from pyspark.sql.types import (
DecimalType, LongType, StringType, StructField, StructType, TimestampType,
)
@pytest.fixture(scope="session")
def spark():
session = (
SparkSession.builder.master("local[2]")
.appName("pyspark-unit-tests")
.config("spark.sql.shuffle.partitions", "2") # tiny data: no need for 200 shuffle tasks
.config("spark.sql.session.timeZone", "UTC") # timestamps don't depend on the machine
.getOrCreate()
)
yield session
session.stop()
ORDERS_SCHEMA = StructType([
StructField("order_id", LongType(), True),
StructField("customer_id", LongType(), True),
StructField("amount", DecimalType(18, 2), True),
StructField("status", StringType(), True),
StructField("event_timestamp", TimestampType(), True),
])
@pytest.fixture
def orders_schema():
return ORDERS_SCHEMA
Every line has a reason:
scope="session": starting the JVM is by far the slowest part of a PySpark test run. One session for the whole run means you pay it once. pytest runs the code afteryieldas teardown when the session ends (pytest fixtures).master("local[2]"): runs Spark in the test process with two worker threads. More than one thread exercises partitioning a little without costing much.spark.sql.shuffle.partitions: the default is 200, which means 200 tasks for a three-rowgroupBy. A small number keeps tests fast.spark.sql.session.timeZone: fixes the zone Spark uses to display and convert timestamps, so a test written in Montréal passes on a CI agent in UTC.session.stop(): releases the JVM cleanly. The pytest example in the Apache Spark testing guide leaves it out; add it (Testing PySpark).
Isolation with a shared session. A shared session is safe as long as tests don’t change shared state. Tests that set Spark configuration at runtime, register temporary views or global UDFs, or cache tables can leak into later tests. Prefer passing DataFrames around; if a test must change configuration, restore it in a try/finally, or give that test its own function-scoped fixture. A function-scoped session is appropriate only for the rare test that needs a different session-level configuration; reusing one session is the norm because creating many is slow.
Running locally on Windows
Spark launches its Python workers with whatever python resolves to. If that’s a different interpreter from the one running pytest (on Windows, often the Microsoft Store alias), tests fail with Python worker failed to connect back. Point Spark at the test interpreter before running pytest:
$env:PYSPARK_PYTHON = (Get-Command python).Source
pytest
Two dependency notes from running this guide’s suite: PySpark needs a supported Java (Spark 4.x requires Java 17 or 21), and with PySpark 3.5, import pyspark.testing also imports pandas-on-Spark, so the test environment needs pandas and pyarrow versions that Spark 3.5 supports, plus setuptools on Python 3.12 or later (it provides the distutils module that 3.5 still imports). Matching your production runtime’s Python version avoids most of this.
The first DataFrame test
Input:
| order_id | customer_id | amount | status |
|---|---|---|---|
| 1 | 10 | 100.00 | paid |
| 2 | 10 | 50.00 | PAID |
| 3 | 20 | 75.00 | cancelled |
Expected:
| customer_id | revenue | paid_orders |
|---|---|---|
| 10 | 150.00 | 2 |
# tests/test_orders.py
from datetime import date, datetime, timedelta, timezone
from decimal import Decimal
import pytest
from pyspark.sql import functions as F
from pyspark.sql.types import (
DateType, DecimalType, DoubleType, LongType, StringType, StructField, StructType,
)
from pyspark.testing import assertDataFrameEqual, assertSchemaEqual
from transforms.orders import (
add_order_date, deduplicate_orders, discard_invalid, enrich_with_segment,
normalize_status, revenue_by_customer, to_eur, transform_orders,
)
T0 = datetime(2026, 11, 9, 12, 0, 0) # fixed time: never datetime.now()
T1 = datetime(2026, 11, 9, 13, 0, 0)
REVENUE_SCHEMA = StructType([
StructField("customer_id", LongType(), True),
StructField("revenue", DecimalType(28, 2), True),
StructField("paid_orders", LongType(), False),
])
def orders(spark, schema, rows):
return spark.createDataFrame(rows, schema)
def test_revenue_counts_paid_orders_per_customer(spark, orders_schema):
source = orders(spark, orders_schema, [
(1, 10, Decimal("100.00"), "paid", T0),
(2, 10, Decimal("50.00"), "PAID", T0),
(3, 20, Decimal("75.00"), "cancelled", T0),
])
expected = spark.createDataFrame([(10, Decimal("150.00"), 2)], REVENUE_SCHEMA)
actual = transform_orders(source)
assertDataFrameEqual(actual, expected)
The test reads as a specification: three orders, one cancelled, two spellings of “paid”, and the revenue that should come out. Note the expected type: summing a decimal(18,2) produces decimal(28,2). That is Spark’s decimal widening rule, and the test documents it instead of hiding it.
(The imports at the top of the file cover the whole suite; the remaining functions come from the same module and are shown with the tests that use them.)
How DataFrame equality works
assertDataFrameEqual(actual, expected) collects both DataFrames and compares them. In the versions tested here:
| Concern | Default behaviour | What still needs care |
|---|---|---|
| Row order | Ignored (checkRowOrder=False) |
Set checkRowOrder=True only when order is part of the contract, such as a sorted export |
| Schema | Compared: column names, column order and types must match | A correct value with the wrong type fails, which is usually what you want |
| Nullability | Ignored: a column that is nullable on one side and not on the other still matches | Assert nullability separately when it is part of a contract |
| Floating point | Compared with tolerances rtol=1e-05, atol=1e-08 |
Decimals are compared exactly; use decimals for money |
| Column order | Compared | PySpark 4.x adds ignoreColumnOrder, ignoreColumnName and ignoreColumnType options; 3.5 has none of them |
| Size | Everything is collected to the driver | Fine for unit tests; never point it at a large table |
The defaults are visible in the signatures (assertDataFrameEqual). The function also accepts lists of Row objects, which is convenient for small results. Prefer a DataFrame with an explicit schema anyway: it checks types, and a list of rows doesn’t.
Two traps that equality won’t catch for you:
- Comparing only counts.
assert actual.count() == 2passes when both rows are wrong. Use counts only where the count is the rule (an empty result, one row per key). - Collecting and comparing lists by hand.
collect()order isn’t guaranteed after a shuffle, soassert actual.collect() == [...]can flake. Use it only on results with one row, or sort explicitly.
Schema testing
def test_revenue_schema_is_the_contract(spark, orders_schema):
source = orders(spark, orders_schema, [(1, 10, Decimal("100.00"), "PAID", T0)])
assertSchemaEqual(transform_orders(source).schema, REVENUE_SCHEMA)
def test_schema_drift_is_caught(spark, orders_schema):
# Someone "simplifies" the aggregation with a cast to double: rows still look right.
source = orders(spark, orders_schema, [(1, 10, Decimal("100.00"), "PAID", T0)])
drifted = transform_orders(source).withColumn("revenue", F.col("revenue").cast("double"))
with pytest.raises(AssertionError):
assertSchemaEqual(drifted.schema, REVENUE_SCHEMA)
Correct rows with the wrong schema can still break downstream consumers.
A revenue that silently becomes double loses decimal precision in a finance report; a customer_id that becomes string breaks every join on it; a timestamp that becomes string sorts alphabetically. When the output schema is part of the contract with consumers, test it explicitly with assertSchemaEqual (assertSchemaEqual). It ignores nullability by default (in 3.5 always, in 4.x through ignoreNullable=True). If a NOT NULL promise is part of the contract, compare field.nullable directly.
The second test is there to show what the first one protects against; in a real suite one schema test per output is enough.
Null handling: encode the business rule
Tests should encode the business rule, not a generic preference.
“Drop nulls” is a preference. “An order without a customer can’t be attributed, but an order without an amount is still a real order” is a rule, and each part of it needs a test:
def test_null_keys_are_discarded(spark, orders_schema):
source = orders(spark, orders_schema, [
(None, 10, Decimal("10.00"), "PAID", T0), # no order_id: can't deduplicate
(2, None, Decimal("20.00"), "PAID", T0), # no customer: can't attribute
(3, 30, Decimal("30.00"), "PAID", T0),
])
assert [r.order_id for r in discard_invalid(source).collect()] == [3]
def test_null_amount_is_kept_but_earns_no_revenue(spark, orders_schema):
source = orders(spark, orders_schema, [
(1, 10, None, "PAID", T0),
(2, 10, Decimal("40.00"), "PAID", T0),
])
assert discard_invalid(source).count() == 2
expected = spark.createDataFrame([(10, Decimal("40.00"), 1)], REVENUE_SCHEMA)
assertDataFrameEqual(transform_orders(source), expected)
def test_null_status_becomes_unknown_not_dropped(spark, orders_schema):
source = orders(spark, orders_schema, [(1, 10, Decimal("5.00"), None, T0)])
assert normalize_status(source).first().status == "UNKNOWN"
The null-amount test is the interesting one: amount >= 0 is null for a null amount, so the filter excludes it without any special code. The test makes that implicit behaviour explicit, so a later refactor that replaces nulls with zero (and suddenly counts the order in paid_orders) fails.
Duplicates and deterministic windows
A replayed event and a genuine status change look the same to distinct(): they don’t. The rule is “latest event per order wins”, which is a window:
def test_latest_event_per_order_wins(spark, orders_schema):
source = orders(spark, orders_schema, [
(1, 10, Decimal("100.00"), "PENDING", T0),
(1, 10, Decimal("100.00"), "PAID", T1), # later event for the same order
(2, 20, Decimal("75.00"), "PAID", T0),
])
expected = orders(spark, orders_schema, [
(1, 10, Decimal("100.00"), "PAID", T1),
(2, 20, Decimal("75.00"), "PAID", T0),
])
assertDataFrameEqual(deduplicate_orders(source), expected)
def test_timestamp_tie_is_broken_deterministically(spark, orders_schema):
source = orders(spark, orders_schema, [
(1, 10, Decimal("100.00"), "PAID", T0),
(1, 10, Decimal("100.00"), "CANCELLED", T0), # same timestamp
])
assert [r.status for r in deduplicate_orders(source).collect()] == ["CANCELLED"]
def test_replayed_event_does_not_double_revenue(spark, orders_schema):
event = (1, 10, Decimal("100.00"), "PAID", T0)
source = orders(spark, orders_schema, [event, event])
expected = spark.createDataFrame([(10, Decimal("100.00"), 1)], REVENUE_SCHEMA)
assertDataFrameEqual(transform_orders(source), expected)
The tie test is the one teams skip. With orderBy(F.col("event_timestamp").desc()) alone, two events with the same timestamp are ranked in whatever order the data arrives in, and the test (and production) can give different answers on different runs. A window used for deduplication needs an ordering that is total for the data it will see; the tie-breaker is a business decision, so write it down and test it. The Window Functions cheat sheet covers the window API itself.
Parameterized tests
One rule, many spellings. Copying four near-identical tests hides the rule in noise and makes it easy to update three of them and forget the fourth. pytest.mark.parametrize runs one test body per value and reports each case separately (parametrizing tests):
@pytest.mark.parametrize("raw", ["paid", "PAID", " Paid", "paid "])
def test_paid_spellings_normalize_to_paid(spark, orders_schema, raw):
source = orders(spark, orders_schema, [(1, 10, Decimal("1.00"), raw, T0)])
assert normalize_status(source).first().status == "PAID"
@pytest.mark.parametrize("raw", ["payed", "", "P A I D", "settled"])
def test_unexpected_statuses_become_unknown(spark, orders_schema, raw):
source = orders(spark, orders_schema, [(1, 10, Decimal("1.00"), raw, T0)])
assert normalize_status(source).first().status == "UNKNOWN"
When the source starts sending Paid\t, the fix is one new value in a list, and the test report names the exact case that failed (test_paid_spellings_normalize_to_paid[ Paid]).
Aggregations: test the grain
def test_one_row_per_customer(spark, orders_schema):
source = orders(spark, orders_schema, [
(1, 10, Decimal("100.00"), "PAID", T0),
(2, 10, Decimal("20.00"), "PAID", T0),
(3, 20, Decimal("5.00"), "PAID", T0),
(4, 30, Decimal("0.00"), "PAID", T0), # zero is a valid amount
])
expected = spark.createDataFrame([
(10, Decimal("120.00"), 2),
(20, Decimal("5.00"), 1),
(30, Decimal("0.00"), 1),
], REVENUE_SCHEMA)
assertDataFrameEqual(transform_orders(source), expected)
Several customers and several orders per customer are what make this a grain test. If someone groups by ("customer_id", "status") or adds order_id to the grouping, customer 10 turns into two rows and the test fails. A one-customer fixture can’t catch that. The GroupBy and Aggregations cheat sheet covers the aggregation API.
Business rules as regression tests
@pytest.mark.parametrize("status", ["CANCELLED", "REFUNDED", "PENDING", "UNKNOWN"])
def test_only_paid_orders_count_as_revenue(spark, orders_schema, status):
source = orders(spark, orders_schema, [(1, 10, Decimal("100.00"), status, T0)])
assert revenue_by_customer(normalize_status(source)).count() == 0
def test_negative_amount_earns_no_revenue(spark, orders_schema):
source = orders(spark, orders_schema, [(1, 10, Decimal("-5.00"), "PAID", T0)])
assert transform_orders(source).count() == 0
Six months from now, someone will be asked to “include refunds in revenue” for one report and will change the shared filter. These tests make that a visible decision instead of a silent one. I checked that they work: changing the filter to F.col("status").isin("PAID", "CANCELLED") in a copy of the module makes exactly two tests fail, test_revenue_counts_paid_orders_per_customer and test_only_paid_orders_count_as_revenue[CANCELLED]. A test you’ve never seen fail hasn’t proven it can.
Empty DataFrames
Empty input is a real case: a source with no orders on a holiday, a partition that filters to nothing. Spark can’t infer column types from zero rows, so an empty DataFrame always needs an explicit schema:
def test_empty_input_gives_empty_output_with_the_same_schema(spark, orders_schema):
empty = spark.createDataFrame([], orders_schema)
actual = transform_orders(empty)
assert actual.count() == 0
assertSchemaEqual(actual.schema, REVENUE_SCHEMA)
def test_empty_dataframe_needs_an_explicit_schema(spark):
with pytest.raises(Exception): # Spark can't infer types from no rows
spark.createDataFrame([])
The first test is the useful one: an empty result with the right schema lets the downstream write create or append to the table correctly. The second only demonstrates why the schema is needed; the exact exception class differs between Spark versions, which is why it catches Exception. Don’t copy that pattern into tests of your own code: there, assert the specific exception.
The same applies to every fixture, not only empty ones: always pass an explicit schema to createDataFrame in tests. Inferred types change with the data: an all-null column has no type, and Python integers become long where production has int.
Exceptions, quarantine and tolerance
Invalid input has three possible designs, and the test follows the design:
| Pattern | When to use it | What the test asserts |
|---|---|---|
| Reject | The input breaks an assumption the code can’t work around | pytest.raises with the specific exception and message |
| Quarantine | Bad rows are expected and must be kept for investigation | Good rows in the output, bad rows in a second DataFrame with a reason |
| Tolerate | The row is valid but incomplete, and the rule says how to treat it | The tolerated behaviour (null amount → no revenue, unknown status → UNKNOWN) |
This guide’s transformation tolerates nulls and unknown statuses. Rejection fits a different kind of problem: a lookup table with duplicate keys, which would silently multiply rows. That leads to joins.
Joins: matched, unmatched and duplicated keys
def enrich_with_segment(orders: DataFrame, customers: DataFrame) -> DataFrame:
"""Add the customer segment. Unmatched orders keep a null segment.
A duplicated customer_id in the lookup would multiply order rows, so it is rejected.
"""
duplicated = customers.groupBy("customer_id").count().where(F.col("count") > 1).limit(1).count()
if duplicated:
raise ValueError("customers has duplicate customer_id values; the join would multiply orders")
return orders.join(customers.select("customer_id", "segment"), on="customer_id", how="left")
def test_join_keeps_unmatched_orders(spark, orders_schema):
source = orders(spark, orders_schema, [
(1, 10, Decimal("100.00"), "PAID", T0),
(2, 99, Decimal("50.00"), "PAID", T0), # customer not in the lookup yet
])
customers = spark.createDataFrame([(10, "Retail")], "customer_id long, segment string")
actual = enrich_with_segment(source, customers).select("order_id", "segment")
expected = spark.createDataFrame([(1, "Retail"), (2, None)], "order_id long, segment string")
assertDataFrameEqual(actual, expected)
def test_duplicate_lookup_key_is_rejected(spark, orders_schema):
source = orders(spark, orders_schema, [(1, 10, Decimal("100.00"), "PAID", T0)])
customers = spark.createDataFrame(
[(10, "Retail"), (10, "Enterprise")], "customer_id long, segment string"
)
with pytest.raises(ValueError, match="duplicate customer_id"):
enrich_with_segment(source, customers)
def test_what_a_duplicate_lookup_key_does_to_a_plain_join(spark, orders_schema):
source = orders(spark, orders_schema, [(1, 10, Decimal("100.00"), "PAID", T0)])
customers = spark.createDataFrame(
[(10, "Retail"), (10, "Enterprise")], "customer_id long, segment string"
)
assert source.join(customers, "customer_id", "left").count() == 2 # one order, two rows
The last test is the reason the guard exists: one order, two lookup rows, two output rows, and the order’s revenue is counted twice downstream. Duplicate keys in a reference table are one of the most common causes of inflated totals, and they come from the data, not the code, so the guard (or an upstream data-quality check) belongs in production as well as in the tests. The Join DataFrames cheat sheet covers join types and cardinality in depth. The guard runs a Spark job, so on large lookups prefer enforcing uniqueness where the lookup is built.
Floating-point values
Money stays decimal. Sometimes a transformation genuinely produces floating-point values (a currency conversion for an indicative report, a ratio, a statistical measure), and exact equality is the wrong test:
def to_eur(orders: DataFrame, rate: float) -> DataFrame:
"""Convert to EUR with a floating-point rate (reporting only; money stays decimal upstream)."""
return orders.withColumn("amount_eur", F.col("amount").cast("double") * F.lit(rate))
def test_currency_conversion_within_tolerance(spark, orders_schema):
source = orders(spark, orders_schema, [(1, 10, Decimal("0.10"), "PAID", T0)])
actual = to_eur(source, rate=0.3).select("order_id", "amount_eur")
expected = spark.createDataFrame([(1, 0.03)], "order_id long, amount_eur double")
# 0.10 * 0.3 is 0.030000000000000002 in binary floating point.
assertDataFrameEqual(actual, expected, rtol=1e-9)
assertDataFrameEqual compares float columns with relative and absolute tolerances (rtol, atol). The defaults (1e-05 and 1e-08) are already loose enough for this case; passing rtol explicitly documents how close is close enough for this calculation. Choose the tolerance from the business meaning (a cent, a basis point), not from whatever makes the test pass.
Timestamps and time zones
Time-dependent tests should freeze the input, not depend on the wall clock.
A test that uses datetime.now() or current_date() passes today and fails at midnight, at month-end, or on the day daylight-saving time changes. Use fixed values (T0 = datetime(2026, 11, 9, 12, 0, 0) above) and put the boundary cases in on purpose:
def add_order_date(orders: DataFrame) -> DataFrame:
"""Business date of the order, in the Spark session time zone."""
return orders.withColumn("order_date", F.to_date("event_timestamp"))
def test_order_date_boundary_at_midnight_utc(spark):
source = spark.createDataFrame(
[(1, "2026-11-09 23:59:59"), (2, "2026-11-10 00:00:00")],
"order_id long, ts string",
).withColumn("event_timestamp", F.to_timestamp("ts")).drop("ts")
actual = add_order_date(source).select("order_id", "order_date")
expected = spark.createDataFrame(
[(1, date(2026, 11, 9)), (2, date(2026, 11, 10))],
StructType([StructField("order_id", LongType()), StructField("order_date", DateType())]),
)
assertDataFrameEqual(actual, expected)
def test_timezone_aware_input_is_unambiguous(spark):
# An aware datetime is one instant, whatever the machine's local time zone.
# (A naive datetime is read in the Python process's local zone, which differs between
# a laptop and a CI agent.) 23:30 at UTC-5 on Nov 9 is 04:30 UTC on Nov 10.
aware = datetime(2026, 11, 9, 23, 30, tzinfo=timezone(timedelta(hours=-5)))
source = spark.createDataFrame([(1, aware)], "order_id long, event_timestamp timestamp")
assert add_order_date(source).first().order_date == date(2026, 11, 10)
Two techniques make timestamp tests portable. Building timestamps from strings with to_timestamp interprets them in the session time zone, which the fixture fixed to UTC. Building them from timezone-aware Python datetimes pins the instant explicitly. Naive Python datetimes are interpreted in the local time zone of the Python process, so a boundary test written with them can pass on a laptop and fail on a CI agent. The naive T0 and T1 used elsewhere are safe only because those tests never derive a date or compare across zones.
Helpers: keep them small
The suite uses one helper, orders(spark, schema, rows), plus shared schemas and fixed timestamps. That’s about the right amount:
- Worth it: a function that builds the input DataFrame with the standard schema; shared
StructTypeconstants; named fixed timestamps. - Not worth it: a custom DataFrame equality function.
assertDataFrameEqualandassertSchemaEqualalready handle row order, schemas and float tolerance, and their failure messages show the differing rows. A home-made comparison (sort, collect, compare lists) is more code and worse diagnostics. - A warning sign: a helper that builds “a typical order” with fifteen defaulted fields. The test then no longer shows which values matter. Prefer writing the three relevant rows out in full.
Mocking: useful at the edges, not for Spark
Mocks replace dependencies you don’t want a unit test to call:
- REST clients that fetch reference data or post results;
- secret and credential wrappers (a Key Vault client,
notebookutils.credentialsin Fabric); - external metadata services, such as a watermark store or a configuration API.
Inject those as arguments (or wrap them in a small function you can patch), and the transformation never knows the difference.
Don’t mock the Spark DataFrame API. A mocked DataFrame whose .filter() returns another mock tests that your code calls filter, not that it filters correctly, and it breaks on every refactor that doesn’t change behaviour. Local Spark is real Spark: it runs the same planner and functions your cluster does. Use it.
I/O boundaries and the Delta integration test
The boundary from the start of this guide, applied: transform_orders() is unit-tested with hand-written DataFrames; read_orders() and write_revenue() are integration-tested against real files or tables with test data.
Read and write are where the storage’s behaviour lives: Delta transactions, schema enforcement on write, overwrite versus append, partition columns, permissions. A unit test that mocks them proves nothing about them; an integration test against real (non-production) storage does.
Delta Lake is the clearest example. A MERGE isn’t a DataFrame transformation; it’s a transactional write against a table, so its tests are integration tests. Here is what one looks like. This example was not run for this guide: it needs the delta-spark package and a Spark session configured for Delta (see the Delta Lake quick start), or a Fabric test workspace where Delta is built in.
# tests/integration/test_orders_delta.py (not run for this guide)
import pytest
from delta.tables import DeltaTable
from pyspark.testing import assertDataFrameEqual
def upsert_orders(spark, batch, path):
target = DeltaTable.forPath(spark, path)
(
target.alias("t")
.merge(batch.alias("s"), "t.order_id = s.order_id")
.whenMatchedUpdateAll()
.whenNotMatchedInsertAll()
.execute()
)
@pytest.mark.integration
def test_rerunning_a_batch_leaves_the_same_state(delta_spark, tmp_path, orders_schema, orders_batch):
path = str(tmp_path / "orders")
delta_spark.createDataFrame([], orders_schema).write.format("delta").save(path)
upsert_orders(delta_spark, orders_batch, path)
after_first_run = delta_spark.read.format("delta").load(path).collect()
upsert_orders(delta_spark, orders_batch, path) # same batch again
after_second_run = delta_spark.read.format("delta").load(path)
assertDataFrameEqual(after_second_run, after_first_run)
delta_spark and orders_batch stand for fixtures you’d define: a Delta-enabled session and a deduplicated batch. pytest’s tmp_path gives each test its own empty directory (temporary directories), and the MERGE API is documented in Delta Lake: upsert into a table using merge. Collecting the first state into rows before the second run keeps the comparison independent of Delta’s versioning.
Idempotency
The test above is an idempotency test: running the same batch twice must leave the same final state, with no duplicated business records. It’s one of the most valuable tests a data pipeline can have, because every pipeline will be retried, and it’s beyond a pure DataFrame unit test, because the property belongs to the write, not to the transformation.
What the unit tests can cover is the part of idempotency that lives in the transformation: test_replayed_event_does_not_double_revenue proves that a duplicated event inside one batch is harmless. Duplicates across batches are handled by the MERGE key and proven by the integration test. Incremental Loads in Microsoft Fabric covers the watermark and MERGE patterns this protects.
Edge cases: which belong in unit tests
| Edge case | Unit test? | Where it’s covered |
|---|---|---|
| Empty DataFrame | Yes | test_empty_input_gives_empty_output_with_the_same_schema |
| One row | Yes | Most tests here use one or two rows |
| Duplicate key | Yes | test_latest_event_per_order_wins, tie-break test |
| Null key | Yes | test_null_keys_are_discarded |
| Zero amount | Yes | test_one_row_per_customer (customer 30) |
| Negative amount | Yes | test_negative_amount_earns_no_revenue |
| Unexpected status | Yes | test_unexpected_statuses_become_unknown |
| Timestamp boundary | Yes | test_order_date_boundary_at_midnight_utc |
| Multiple customers | Yes | test_one_row_per_customer (grain) |
| Duplicated event | Yes, within a batch | test_replayed_event_does_not_double_revenue |
| Duplicated batch or file | No: integration | The Delta idempotency test |
| Real null rates, volumes, distributions | No: data quality | Production data-quality checks on every load |
| Schema drift at the source | Partly | Unit tests pin the output schema; contract checks on the source belong to integration and data quality |
The split follows the boundary from the start: a unit test proves what the code does with a row; whether the rows that arrive in production look like that is a data-quality question, answered by checks on real data (Testing a Production Data Platform).
Keep unit tests fast
- Small DataFrames. Rows written in the test, not sampled from a file.
- One shared SparkSession. JVM startup dominates; pay it once.
- Few shuffle partitions. The default 200 is tuned for clusters, not three rows.
- No filesystem or network. Those belong to integration tests, which can be slower because they run less often.
- Avoid unnecessary actions. Each
count(),collect()orfirst()runs a Spark job; assert once per test where possible.
There’s no universal time budget. The goal is that developers run the suite before every push without thinking about it. On the Windows laptop used for this guide, most of the 82 seconds for 31 tests went into Spark and Python-worker startup and per-job overhead, not into the data; a Linux CI agent is usually faster. Measure your own suite and watch for it growing.
Where unit tests sit in CI
- Pull request
- Python checks
- PySpark unit testspytest, local Spark
- Build
- Deploy to Test
- Integration testspytest -m integration
The unit tests need only Python, Java and the PySpark package, so any CI agent can run them on every pull request. Integration tests run after deployment, against a test environment, with pytest -m integration. Testing a Production Data Platform covers the full gate sequence.
Coverage: supporting evidence, not the goal
Line coverage tells you which Python lines ran during the tests. For PySpark code it is weaker than usual, because most of the logic is inside Spark expressions: revenue_by_customer is “100% covered” by any test that calls it once, whether or not a cancelled order, a null amount or a second customer was ever in the input.
High coverage does not prove:
- that the transformation is correct for the edge cases that matter;
- that the output schema matches its contract;
- that production data looks like the fixtures (data quality);
- that the job behaves on real storage or at volume (integration and performance).
Use coverage to find functions no test calls at all. Judge the suite by the edge-case table above.
What belongs in Fabric tests
The same boundaries map onto Microsoft Fabric:
| Level | What it tests | Where |
|---|---|---|
| Local unit tests | Transformation logic, schema logic, business rules | pytest and local Spark, on a laptop or CI agent; the library is then attached to a Fabric environment or imported by the notebook |
| Fabric integration tests | Notebook parameters, Lakehouse reads and writes, workspace identity, pipeline execution, environment configuration | A test workspace with a test Lakehouse, triggered from CI |
| Production validation | Data quality, performance, security, monitoring | Production, on every load and after every release |
Microsoft Fabric Notebook Parameters shows how to keep notebook logic in functions that can be tested this way, and the flagship’s Microsoft Fabric testing section covers the integration and production layers.
Common mistakes
| Mistake | Do this instead |
|---|---|
| Huge test datasets | A few designed rows per behaviour |
| Production data in tests | Hand-written fixtures; nothing to mask, nothing to leak |
| Comparing only row counts | assertDataFrameEqual against an expected DataFrame |
| Ignoring the schema | Explicit input schemas; assertSchemaEqual on outputs that are contracts |
| Relying on row order | Let assertDataFrameEqual ignore order; make window orderings total |
collect() on anything big, or comparing collected lists after a shuffle |
Collect only tiny, single-row results; compare DataFrames otherwise |
| One giant end-to-end test | Unit tests per rule, plus a few integration tests |
| Testing Spark itself | Test your rule (latest event wins), not that row_number() numbers rows |
now() and current_date() in tests |
Fixed timestamps, explicit session time zone |
| Shared mutable test data | Build input inside each test; schemas and constants can be shared |
| Accepting flaky Spark tests | Find the cause (ordering, time, shared state) and fix it; a test that is sometimes red is a test nobody trusts |
Before calling a PySpark transformation tested
- Happy path
- Empty input, with the output schema
- Null behaviour, per column, as a business rule
- Duplicate behaviour, including ties
- Output schema
- Edge values (zero, negative, unexpected category)
- Join behaviour (unmatched, duplicated lookup key)
- Aggregation grain (several groups, several rows per group)
- Deterministic timestamps and boundaries
- Expected error behaviour
- Regression tests for the business rules that matter most
Complete example
One transformation and its whole test file, ready to copy into a project (with the pyproject.toml from the layout section). It brings together the Spark fixture, explicit schemas, a test input with a later cancellation and a missing customer, the expected output, a schema assertion and the empty-input edge case.
# src/transforms/paid_revenue.py
from pyspark.sql import DataFrame, Window
from pyspark.sql import functions as F
def paid_revenue(orders: DataFrame) -> DataFrame:
"""Revenue per customer from the latest event of each PAID order."""
latest = Window.partitionBy("order_id").orderBy(F.col("event_timestamp").desc(), F.col("status").asc())
return (
orders.withColumn("status", F.upper(F.trim("status")))
.where(F.col("order_id").isNotNull() & F.col("customer_id").isNotNull())
.withColumn("_rank", F.row_number().over(latest))
.where((F.col("_rank") == 1) & (F.col("status") == "PAID"))
.groupBy("customer_id")
.agg(F.sum("amount").alias("revenue"))
)
# tests/test_paid_revenue.py
from datetime import datetime
from decimal import Decimal
import pytest
from pyspark.sql import SparkSession
from pyspark.sql.types import (
DecimalType, LongType, StringType, StructField, StructType, TimestampType,
)
from pyspark.testing import assertDataFrameEqual, assertSchemaEqual
from transforms.paid_revenue import paid_revenue
ORDERS = StructType([
StructField("order_id", LongType()),
StructField("customer_id", LongType()),
StructField("amount", DecimalType(18, 2)),
StructField("status", StringType()),
StructField("event_timestamp", TimestampType()),
])
REVENUE = StructType([
StructField("customer_id", LongType()),
StructField("revenue", DecimalType(28, 2)),
])
T0, T1 = datetime(2026, 11, 9, 12, 0), datetime(2026, 11, 9, 13, 0)
@pytest.fixture(scope="session")
def spark():
session = (
SparkSession.builder.master("local[2]")
.appName("paid-revenue-tests")
.config("spark.sql.shuffle.partitions", "2")
.config("spark.sql.session.timeZone", "UTC")
.getOrCreate()
)
yield session
session.stop()
def test_paid_revenue(spark):
orders = spark.createDataFrame([
(1, 10, Decimal("100.00"), "paid", T0),
(2, 10, Decimal("50.00"), " PAID", T0),
(3, 20, Decimal("75.00"), "PAID", T0),
(3, 20, Decimal("75.00"), "cancelled", T1), # later event cancels order 3
(4, None, Decimal("10.00"), "PAID", T0), # no customer: not attributable
], ORDERS)
expected = spark.createDataFrame([(10, Decimal("150.00"))], REVENUE)
actual = paid_revenue(orders)
assertSchemaEqual(actual.schema, expected.schema)
assertDataFrameEqual(actual, expected)
def test_empty_input(spark):
actual = paid_revenue(spark.createDataFrame([], ORDERS))
assert actual.count() == 0
assertSchemaEqual(actual.schema, REVENUE)
Both tests pass on PySpark 3.5.9. Customer 20 disappears because order 3’s latest event is a cancellation, and order 4 is dropped because it has no customer: two rules, proven with five rows.
What to read next
- Testing a Production Data Platform: where unit tests sit among data-quality checks, integration tests and CI/CD gates.
- Incremental Loads in Microsoft Fabric: the watermark and
MERGEpatterns that idempotency tests protect. - Microsoft Fabric Notebook Parameters: keeping notebook logic testable.
- Explore Data with PySpark in Microsoft Fabric: the DataFrame basics these tests build on.
- Cheat sheets: Join DataFrames in PySpark, Window Functions, GroupBy and Aggregations, Data Quality Checks.
Sources
- PySpark testing: Testing PySpark (official guide, pytest and unittest patterns),
assertDataFrameEqual,assertSchemaEqual. Signatures and defaults were also read from the installed PySpark 3.5.9 and 4.3.0 packages. - pytest: Fixtures, Parametrizing tests, Temporary directories and files, Marks
- Delta Lake: Quick start, Upsert into a table using merge
- Microsoft Fabric: Apache Spark runtimes in Fabric (Spark, Python and Java versions per runtime)
Last verified: November 2026. The unit tests and the complete example were executed with PySpark 3.5.9, pytest 9.2.0, Python 3.12.10 and Java 21 (31 and 2 tests, all passing); the Delta integration example was not executed. The testing boundaries, checklist and patterns are Javi Builds guidance; API behaviour comes from the sources above.
¿Te resultó útil este contenido?
¿Qué podríamos mejorar?
Gracias — esto me ayuda a mejorar Javi Builds.
Sigue aprendiendo
- Guía rápidaLeer y escribir tablas DeltaLee y escribe tablas Delta con PySpark usando patrones seguros de append, overwrite, merge, evolución del esquema, particionamiento y mantenimiento para pipelines de datos en producción.Abrir guía rápida
- RecursoContenido en inglésCustomer DatasetA synthetic customer master dataset for practicing CSV ingestion, inspection, cleaning, filtering and joins with pandas.Ver recurso
¿Qué debería crear después?
Estoy construyendo Javi Builds alrededor de problemas reales que enfrentan los ingenieros de datos. Dime qué tema te gustaría ver después.
Mantente al día
Recibe solo las novedades de Javi Builds que realmente te interesan.
Revisa tu bandeja de entrada
Te he enviado un enlace de confirmación. Tu suscripción se activará cuando la confirmes.