Python

PySpark DataFrame Unit Testing with pytest: A Practical Guide

Learn how to unit test PySpark DataFrame transformations with pytest using small deterministic datasets, schema assertions, fixtures, edge cases and reusable testing patterns, and where unit tests stop and integration tests begin.

Par JaviPublié 24 min de lectureIntermédiaire

Dans cet article

This guide answers one question:

How do I prove that a PySpark transformation is correct without running an entire production pipeline?

The answer is a unit test: a few rows in, the transformation, the expected rows out, on a local Spark session, in seconds. Three principles shape everything below:

  • Test transformation logic with the smallest DataFrame that proves the behaviour.
  • A Spark job doesn’t need millions of rows to prove that its business logic is correct. It needs the right three rows: the duplicate, the null, the cancelled order.
  • Unit tests isolate transformation behaviour. Storage, orchestration and infrastructure belong in integration tests.

This is the hands-on companion to Testing a Production Data Platform, which explains where unit tests sit among data-quality checks, integration tests and CI/CD gates. Nothing here is specific to Microsoft Fabric; a short section at the end maps the boundaries onto Fabric.

Unit, integration and end-to-end: draw the boundary first

Most PySpark testing pain comes from a test that tries to be all three kinds at once: it reads a table, transforms it, writes it, and then nobody can tell which part broke.

  • Unit testThe logic, in isolation
    1. Input DataFrameA few rows, written in the test
    2. TransformationA Python function
    3. Seconds, local SparkExpected DataFrame
  • Integration testSpark plus real storage
    1. Read Delta
    2. Transform
    3. Real tables, test dataWrite Delta
  • End-to-end testThe whole pipeline
    1. Source
    2. Pipeline and notebook
    3. Slow, few, criticalLakehouse and Warehouse
Three kinds of PySpark test. Each kind proves something the others can't. A unit test proves the logic in seconds; an integration test proves Spark works with real storage; an end-to-end test proves the pipeline is wired correctly. Keep them in separate test runs.
Kind Example What it proves Where it runs
Unit input DataFrame → transform_orders → expected DataFrame The business logic is right for the cases you designed Developer machine, every pull request
Integration read Delta → transform → write Delta, a MERGE run twice Spark, Delta and the storage behave as the code assumes A test environment with real storage
End-to-end source → Fabric pipeline → notebook → Lakehouse → Warehouse The pieces are connected, configured and ordered correctly A test workspace, before release

This guide is about the first row, and about keeping the other two out of it.

The example: orders

Every example uses the same small domain, so the tests read as a specification of one transformation.

Column Type Meaning
order_id long Business key; an order can have several events
customer_id long Who placed it
amount decimal(18,2) Order value; money stays decimal
status string paid, PAID , cancelled… as sent by the source
event_timestamp timestamp When this version of the order happened

The business rules, written down before any test:

  1. Normalize status: trim and upper-case. Known values are PAID, PENDING, CANCELLED, REFUNDED; null or anything else becomes UNKNOWN. Unknown rows are kept, so they stay visible.
  2. Discard invalid rows: an order needs an order_id and a customer_id to be attributed. Nothing else is discarded.
  3. Deduplicate: keep the latest event per order_id. On a timestamp tie, status ascending decides, so CANCELLED beats PAID.
  4. Revenue: only PAID orders with a known, non-negative amount count. Aggregate per customer: revenue and paid_orders.

Structure the code so it can be tested

Notebook code often looks like this:

df = spark.read.table("bronze.orders")
df = df.withColumn("status", F.upper(F.trim("status")))
df = df.where(F.col("customer_id").isNotNull())
# ... forty more lines ...
df.write.mode("overwrite").saveAsTable("gold.customer_revenue")

To test the logic you would need the bronze.orders table, write access to gold, and a cleanup step, and a failure could come from any of the three. Separate the parts instead:

def read_orders(spark):
    return spark.read.table("bronze.orders")


def write_revenue(revenue):
    revenue.write.mode("overwrite").saveAsTable("gold.customer_revenue")


# Orchestration: the notebook or job entry point
revenue = transform_orders(read_orders(spark))
write_revenue(revenue)

transform_orders takes a DataFrame and returns a DataFrame. It has no spark.read, no write, no paths and no table names, which is what makes it unit-testable.

Keep transformation functions as pure as practical.

Spark transformations aren’t pure in the strict functional sense: they build a lazy plan that runs on a cluster, and functions such as current_timestamp() change between runs. The useful version of the rule is narrower: a transformation function receives every input as an argument and does no I/O. That alone makes it possible to call it from a test with a hand-written DataFrame.

  1. Integration testread_orders()Real table or files
  2. Unit testtransform_orders()DataFrame in, DataFrame out
  3. Integration testwrite_revenue()Real table, real format
Where each test type applies. Read and write are integration-tested against real storage; the transformation in the middle is unit-tested with hand-written DataFrames.

Project layout

A small layout is enough:

pyproject.toml
src/
  transforms/
    __init__.py
    orders.py
tests/
  conftest.py
  test_orders.py

pytest needs to import transforms from src. The simplest way is pytest’s pythonpath setting:

# pyproject.toml
[tool.pytest.ini_options]
pythonpath = ["src"]
testpaths = ["tests"]
markers = ["integration: needs real storage (Delta, a Lakehouse); excluded from the unit-test run"]
addopts = "-m 'not integration'"

The alternative, when the transformations ship as a wheel (for example as a custom library in a Fabric environment), is to install the package in editable mode (pip install -e .) and drop pythonpath. Either works; pick the one that matches how the code is deployed. The integration marker and addopts keep integration tests out of the default run; they come back in a separate CI step (pytest markers).

Pin PySpark to the Spark version you deploy on. Testing against Spark 4.x and running on Spark 3.5 (or the reverse) lets version differences through. In Microsoft Fabric, Runtime 2.0 runs Spark 4.1 with Python 3.13, and Runtime 1.3 runs Spark 3.5.5 with Python 3.11 (Fabric runtimes).

The transformation under test

The core of the module, exactly as tested (three smaller helpers used later, add_order_date, to_eur and enrich_with_segment, live in the same file and are shown with their tests):

# src/transforms/orders.py
"""Order transformations: pure DataFrame-in, DataFrame-out functions with no I/O."""
from pyspark.sql import DataFrame, Window
from pyspark.sql import functions as F

KNOWN_STATUSES = ["PAID", "PENDING", "CANCELLED", "REFUNDED"]


def normalize_status(orders: DataFrame) -> DataFrame:
    """Trim and upper-case status. Null or unknown values become UNKNOWN (kept, not dropped)."""
    cleaned = F.upper(F.trim(F.col("status")))
    return orders.withColumn(
        "status",
        F.when(cleaned.isin(KNOWN_STATUSES), cleaned).otherwise(F.lit("UNKNOWN")),
    )


def discard_invalid(orders: DataFrame) -> DataFrame:
    """An order needs an order_id and a customer_id to be attributed. Everything else is kept."""
    return orders.where(F.col("order_id").isNotNull() & F.col("customer_id").isNotNull())


def deduplicate_orders(orders: DataFrame) -> DataFrame:
    """Keep the latest event per order_id.

    Tie-break (same event_timestamp): status ascending, so CANCELLED wins over PAID.
    The tie-break is a business decision; it is written down so the result is deterministic.
    """
    latest_first = Window.partitionBy("order_id").orderBy(
        F.col("event_timestamp").desc(), F.col("status").asc()
    )
    return (
        orders.withColumn("_rank", F.row_number().over(latest_first))
        .where(F.col("_rank") == 1)
        .drop("_rank")
    )


def revenue_by_customer(orders: DataFrame) -> DataFrame:
    """Revenue counts PAID orders only, with a known, non-negative amount."""
    return (
        orders.where((F.col("status") == "PAID") & (F.col("amount") >= 0))
        .groupBy("customer_id")
        .agg(
            F.sum("amount").alias("revenue"),
            F.count("*").alias("paid_orders"),
        )
    )


def transform_orders(orders: DataFrame) -> DataFrame:
    """The full transformation: normalize, validate, deduplicate, aggregate."""
    return revenue_by_customer(deduplicate_orders(discard_invalid(normalize_status(orders))))

Small functions composed into transform_orders give you two levels of test for free: each rule on its own, and the whole transformation end to end on a handful of rows.

The SparkSession fixture

# tests/conftest.py
import pytest
from pyspark.sql import SparkSession
from pyspark.sql.types import (
    DecimalType, LongType, StringType, StructField, StructType, TimestampType,
)


@pytest.fixture(scope="session")
def spark():
    session = (
        SparkSession.builder.master("local[2]")
        .appName("pyspark-unit-tests")
        .config("spark.sql.shuffle.partitions", "2")   # tiny data: no need for 200 shuffle tasks
        .config("spark.sql.session.timeZone", "UTC")   # timestamps don't depend on the machine
        .getOrCreate()
    )
    yield session
    session.stop()


ORDERS_SCHEMA = StructType([
    StructField("order_id", LongType(), True),
    StructField("customer_id", LongType(), True),
    StructField("amount", DecimalType(18, 2), True),
    StructField("status", StringType(), True),
    StructField("event_timestamp", TimestampType(), True),
])


@pytest.fixture
def orders_schema():
    return ORDERS_SCHEMA

Every line has a reason:

  • scope="session": starting the JVM is by far the slowest part of a PySpark test run. One session for the whole run means you pay it once. pytest runs the code after yield as teardown when the session ends (pytest fixtures).
  • master("local[2]"): runs Spark in the test process with two worker threads. More than one thread exercises partitioning a little without costing much.
  • spark.sql.shuffle.partitions: the default is 200, which means 200 tasks for a three-row groupBy. A small number keeps tests fast.
  • spark.sql.session.timeZone: fixes the zone Spark uses to display and convert timestamps, so a test written in Montréal passes on a CI agent in UTC.
  • session.stop(): releases the JVM cleanly. The pytest example in the Apache Spark testing guide leaves it out; add it (Testing PySpark).

Isolation with a shared session. A shared session is safe as long as tests don’t change shared state. Tests that set Spark configuration at runtime, register temporary views or global UDFs, or cache tables can leak into later tests. Prefer passing DataFrames around; if a test must change configuration, restore it in a try/finally, or give that test its own function-scoped fixture. A function-scoped session is appropriate only for the rare test that needs a different session-level configuration; reusing one session is the norm because creating many is slow.

Running locally on Windows

Spark launches its Python workers with whatever python resolves to. If that’s a different interpreter from the one running pytest (on Windows, often the Microsoft Store alias), tests fail with Python worker failed to connect back. Point Spark at the test interpreter before running pytest:

$env:PYSPARK_PYTHON = (Get-Command python).Source
pytest

Two dependency notes from running this guide’s suite: PySpark needs a supported Java (Spark 4.x requires Java 17 or 21), and with PySpark 3.5, import pyspark.testing also imports pandas-on-Spark, so the test environment needs pandas and pyarrow versions that Spark 3.5 supports, plus setuptools on Python 3.12 or later (it provides the distutils module that 3.5 still imports). Matching your production runtime’s Python version avoids most of this.

The first DataFrame test

Input:

order_id customer_id amount status
1 10 100.00 paid
2 10 50.00 PAID
3 20 75.00 cancelled

Expected:

customer_id revenue paid_orders
10 150.00 2
# tests/test_orders.py
from datetime import date, datetime, timedelta, timezone
from decimal import Decimal

import pytest
from pyspark.sql import functions as F
from pyspark.sql.types import (
    DateType, DecimalType, DoubleType, LongType, StringType, StructField, StructType,
)
from pyspark.testing import assertDataFrameEqual, assertSchemaEqual

from transforms.orders import (
    add_order_date, deduplicate_orders, discard_invalid, enrich_with_segment,
    normalize_status, revenue_by_customer, to_eur, transform_orders,
)

T0 = datetime(2026, 11, 9, 12, 0, 0)          # fixed time: never datetime.now()
T1 = datetime(2026, 11, 9, 13, 0, 0)

REVENUE_SCHEMA = StructType([
    StructField("customer_id", LongType(), True),
    StructField("revenue", DecimalType(28, 2), True),
    StructField("paid_orders", LongType(), False),
])


def orders(spark, schema, rows):
    return spark.createDataFrame(rows, schema)


def test_revenue_counts_paid_orders_per_customer(spark, orders_schema):
    source = orders(spark, orders_schema, [
        (1, 10, Decimal("100.00"), "paid", T0),
        (2, 10, Decimal("50.00"), "PAID", T0),
        (3, 20, Decimal("75.00"), "cancelled", T0),
    ])
    expected = spark.createDataFrame([(10, Decimal("150.00"), 2)], REVENUE_SCHEMA)

    actual = transform_orders(source)

    assertDataFrameEqual(actual, expected)

The test reads as a specification: three orders, one cancelled, two spellings of “paid”, and the revenue that should come out. Note the expected type: summing a decimal(18,2) produces decimal(28,2). That is Spark’s decimal widening rule, and the test documents it instead of hiding it.

(The imports at the top of the file cover the whole suite; the remaining functions come from the same module and are shown with the tests that use them.)

How DataFrame equality works

assertDataFrameEqual(actual, expected) collects both DataFrames and compares them. In the versions tested here:

Concern Default behaviour What still needs care
Row order Ignored (checkRowOrder=False) Set checkRowOrder=True only when order is part of the contract, such as a sorted export
Schema Compared: column names, column order and types must match A correct value with the wrong type fails, which is usually what you want
Nullability Ignored: a column that is nullable on one side and not on the other still matches Assert nullability separately when it is part of a contract
Floating point Compared with tolerances rtol=1e-05, atol=1e-08 Decimals are compared exactly; use decimals for money
Column order Compared PySpark 4.x adds ignoreColumnOrder, ignoreColumnName and ignoreColumnType options; 3.5 has none of them
Size Everything is collected to the driver Fine for unit tests; never point it at a large table

The defaults are visible in the signatures (assertDataFrameEqual). The function also accepts lists of Row objects, which is convenient for small results. Prefer a DataFrame with an explicit schema anyway: it checks types, and a list of rows doesn’t.

Two traps that equality won’t catch for you:

  • Comparing only counts. assert actual.count() == 2 passes when both rows are wrong. Use counts only where the count is the rule (an empty result, one row per key).
  • Collecting and comparing lists by hand. collect() order isn’t guaranteed after a shuffle, so assert actual.collect() == [...] can flake. Use it only on results with one row, or sort explicitly.

Schema testing

def test_revenue_schema_is_the_contract(spark, orders_schema):
    source = orders(spark, orders_schema, [(1, 10, Decimal("100.00"), "PAID", T0)])
    assertSchemaEqual(transform_orders(source).schema, REVENUE_SCHEMA)


def test_schema_drift_is_caught(spark, orders_schema):
    # Someone "simplifies" the aggregation with a cast to double: rows still look right.
    source = orders(spark, orders_schema, [(1, 10, Decimal("100.00"), "PAID", T0)])
    drifted = transform_orders(source).withColumn("revenue", F.col("revenue").cast("double"))
    with pytest.raises(AssertionError):
        assertSchemaEqual(drifted.schema, REVENUE_SCHEMA)

Correct rows with the wrong schema can still break downstream consumers.

A revenue that silently becomes double loses decimal precision in a finance report; a customer_id that becomes string breaks every join on it; a timestamp that becomes string sorts alphabetically. When the output schema is part of the contract with consumers, test it explicitly with assertSchemaEqual (assertSchemaEqual). It ignores nullability by default (in 3.5 always, in 4.x through ignoreNullable=True). If a NOT NULL promise is part of the contract, compare field.nullable directly.

The second test is there to show what the first one protects against; in a real suite one schema test per output is enough.

Null handling: encode the business rule

Tests should encode the business rule, not a generic preference.

“Drop nulls” is a preference. “An order without a customer can’t be attributed, but an order without an amount is still a real order” is a rule, and each part of it needs a test:

def test_null_keys_are_discarded(spark, orders_schema):
    source = orders(spark, orders_schema, [
        (None, 10, Decimal("10.00"), "PAID", T0),   # no order_id: can't deduplicate
        (2, None, Decimal("20.00"), "PAID", T0),    # no customer: can't attribute
        (3, 30, Decimal("30.00"), "PAID", T0),
    ])
    assert [r.order_id for r in discard_invalid(source).collect()] == [3]


def test_null_amount_is_kept_but_earns_no_revenue(spark, orders_schema):
    source = orders(spark, orders_schema, [
        (1, 10, None, "PAID", T0),
        (2, 10, Decimal("40.00"), "PAID", T0),
    ])
    assert discard_invalid(source).count() == 2
    expected = spark.createDataFrame([(10, Decimal("40.00"), 1)], REVENUE_SCHEMA)
    assertDataFrameEqual(transform_orders(source), expected)


def test_null_status_becomes_unknown_not_dropped(spark, orders_schema):
    source = orders(spark, orders_schema, [(1, 10, Decimal("5.00"), None, T0)])
    assert normalize_status(source).first().status == "UNKNOWN"

The null-amount test is the interesting one: amount >= 0 is null for a null amount, so the filter excludes it without any special code. The test makes that implicit behaviour explicit, so a later refactor that replaces nulls with zero (and suddenly counts the order in paid_orders) fails.

Duplicates and deterministic windows

A replayed event and a genuine status change look the same to distinct(): they don’t. The rule is “latest event per order wins”, which is a window:

def test_latest_event_per_order_wins(spark, orders_schema):
    source = orders(spark, orders_schema, [
        (1, 10, Decimal("100.00"), "PENDING", T0),
        (1, 10, Decimal("100.00"), "PAID", T1),     # later event for the same order
        (2, 20, Decimal("75.00"), "PAID", T0),
    ])
    expected = orders(spark, orders_schema, [
        (1, 10, Decimal("100.00"), "PAID", T1),
        (2, 20, Decimal("75.00"), "PAID", T0),
    ])
    assertDataFrameEqual(deduplicate_orders(source), expected)


def test_timestamp_tie_is_broken_deterministically(spark, orders_schema):
    source = orders(spark, orders_schema, [
        (1, 10, Decimal("100.00"), "PAID", T0),
        (1, 10, Decimal("100.00"), "CANCELLED", T0),   # same timestamp
    ])
    assert [r.status for r in deduplicate_orders(source).collect()] == ["CANCELLED"]


def test_replayed_event_does_not_double_revenue(spark, orders_schema):
    event = (1, 10, Decimal("100.00"), "PAID", T0)
    source = orders(spark, orders_schema, [event, event])
    expected = spark.createDataFrame([(10, Decimal("100.00"), 1)], REVENUE_SCHEMA)
    assertDataFrameEqual(transform_orders(source), expected)

The tie test is the one teams skip. With orderBy(F.col("event_timestamp").desc()) alone, two events with the same timestamp are ranked in whatever order the data arrives in, and the test (and production) can give different answers on different runs. A window used for deduplication needs an ordering that is total for the data it will see; the tie-breaker is a business decision, so write it down and test it. The Window Functions cheat sheet covers the window API itself.

Parameterized tests

One rule, many spellings. Copying four near-identical tests hides the rule in noise and makes it easy to update three of them and forget the fourth. pytest.mark.parametrize runs one test body per value and reports each case separately (parametrizing tests):

@pytest.mark.parametrize("raw", ["paid", "PAID", " Paid", "paid "])
def test_paid_spellings_normalize_to_paid(spark, orders_schema, raw):
    source = orders(spark, orders_schema, [(1, 10, Decimal("1.00"), raw, T0)])
    assert normalize_status(source).first().status == "PAID"


@pytest.mark.parametrize("raw", ["payed", "", "P A I D", "settled"])
def test_unexpected_statuses_become_unknown(spark, orders_schema, raw):
    source = orders(spark, orders_schema, [(1, 10, Decimal("1.00"), raw, T0)])
    assert normalize_status(source).first().status == "UNKNOWN"

When the source starts sending Paid\t, the fix is one new value in a list, and the test report names the exact case that failed (test_paid_spellings_normalize_to_paid[ Paid]).

Aggregations: test the grain

def test_one_row_per_customer(spark, orders_schema):
    source = orders(spark, orders_schema, [
        (1, 10, Decimal("100.00"), "PAID", T0),
        (2, 10, Decimal("20.00"), "PAID", T0),
        (3, 20, Decimal("5.00"), "PAID", T0),
        (4, 30, Decimal("0.00"), "PAID", T0),       # zero is a valid amount
    ])
    expected = spark.createDataFrame([
        (10, Decimal("120.00"), 2),
        (20, Decimal("5.00"), 1),
        (30, Decimal("0.00"), 1),
    ], REVENUE_SCHEMA)
    assertDataFrameEqual(transform_orders(source), expected)

Several customers and several orders per customer are what make this a grain test. If someone groups by ("customer_id", "status") or adds order_id to the grouping, customer 10 turns into two rows and the test fails. A one-customer fixture can’t catch that. The GroupBy and Aggregations cheat sheet covers the aggregation API.

Business rules as regression tests

@pytest.mark.parametrize("status", ["CANCELLED", "REFUNDED", "PENDING", "UNKNOWN"])
def test_only_paid_orders_count_as_revenue(spark, orders_schema, status):
    source = orders(spark, orders_schema, [(1, 10, Decimal("100.00"), status, T0)])
    assert revenue_by_customer(normalize_status(source)).count() == 0


def test_negative_amount_earns_no_revenue(spark, orders_schema):
    source = orders(spark, orders_schema, [(1, 10, Decimal("-5.00"), "PAID", T0)])
    assert transform_orders(source).count() == 0

Six months from now, someone will be asked to “include refunds in revenue” for one report and will change the shared filter. These tests make that a visible decision instead of a silent one. I checked that they work: changing the filter to F.col("status").isin("PAID", "CANCELLED") in a copy of the module makes exactly two tests fail, test_revenue_counts_paid_orders_per_customer and test_only_paid_orders_count_as_revenue[CANCELLED]. A test you’ve never seen fail hasn’t proven it can.

Empty DataFrames

Empty input is a real case: a source with no orders on a holiday, a partition that filters to nothing. Spark can’t infer column types from zero rows, so an empty DataFrame always needs an explicit schema:

def test_empty_input_gives_empty_output_with_the_same_schema(spark, orders_schema):
    empty = spark.createDataFrame([], orders_schema)
    actual = transform_orders(empty)
    assert actual.count() == 0
    assertSchemaEqual(actual.schema, REVENUE_SCHEMA)


def test_empty_dataframe_needs_an_explicit_schema(spark):
    with pytest.raises(Exception):              # Spark can't infer types from no rows
        spark.createDataFrame([])

The first test is the useful one: an empty result with the right schema lets the downstream write create or append to the table correctly. The second only demonstrates why the schema is needed; the exact exception class differs between Spark versions, which is why it catches Exception. Don’t copy that pattern into tests of your own code: there, assert the specific exception.

The same applies to every fixture, not only empty ones: always pass an explicit schema to createDataFrame in tests. Inferred types change with the data: an all-null column has no type, and Python integers become long where production has int.

Exceptions, quarantine and tolerance

Invalid input has three possible designs, and the test follows the design:

Pattern When to use it What the test asserts
Reject The input breaks an assumption the code can’t work around pytest.raises with the specific exception and message
Quarantine Bad rows are expected and must be kept for investigation Good rows in the output, bad rows in a second DataFrame with a reason
Tolerate The row is valid but incomplete, and the rule says how to treat it The tolerated behaviour (null amount → no revenue, unknown status → UNKNOWN)

This guide’s transformation tolerates nulls and unknown statuses. Rejection fits a different kind of problem: a lookup table with duplicate keys, which would silently multiply rows. That leads to joins.

Joins: matched, unmatched and duplicated keys

def enrich_with_segment(orders: DataFrame, customers: DataFrame) -> DataFrame:
    """Add the customer segment. Unmatched orders keep a null segment.

    A duplicated customer_id in the lookup would multiply order rows, so it is rejected.
    """
    duplicated = customers.groupBy("customer_id").count().where(F.col("count") > 1).limit(1).count()
    if duplicated:
        raise ValueError("customers has duplicate customer_id values; the join would multiply orders")
    return orders.join(customers.select("customer_id", "segment"), on="customer_id", how="left")
def test_join_keeps_unmatched_orders(spark, orders_schema):
    source = orders(spark, orders_schema, [
        (1, 10, Decimal("100.00"), "PAID", T0),
        (2, 99, Decimal("50.00"), "PAID", T0),      # customer not in the lookup yet
    ])
    customers = spark.createDataFrame([(10, "Retail")], "customer_id long, segment string")
    actual = enrich_with_segment(source, customers).select("order_id", "segment")
    expected = spark.createDataFrame([(1, "Retail"), (2, None)], "order_id long, segment string")
    assertDataFrameEqual(actual, expected)


def test_duplicate_lookup_key_is_rejected(spark, orders_schema):
    source = orders(spark, orders_schema, [(1, 10, Decimal("100.00"), "PAID", T0)])
    customers = spark.createDataFrame(
        [(10, "Retail"), (10, "Enterprise")], "customer_id long, segment string"
    )
    with pytest.raises(ValueError, match="duplicate customer_id"):
        enrich_with_segment(source, customers)


def test_what_a_duplicate_lookup_key_does_to_a_plain_join(spark, orders_schema):
    source = orders(spark, orders_schema, [(1, 10, Decimal("100.00"), "PAID", T0)])
    customers = spark.createDataFrame(
        [(10, "Retail"), (10, "Enterprise")], "customer_id long, segment string"
    )
    assert source.join(customers, "customer_id", "left").count() == 2   # one order, two rows

The last test is the reason the guard exists: one order, two lookup rows, two output rows, and the order’s revenue is counted twice downstream. Duplicate keys in a reference table are one of the most common causes of inflated totals, and they come from the data, not the code, so the guard (or an upstream data-quality check) belongs in production as well as in the tests. The Join DataFrames cheat sheet covers join types and cardinality in depth. The guard runs a Spark job, so on large lookups prefer enforcing uniqueness where the lookup is built.

Floating-point values

Money stays decimal. Sometimes a transformation genuinely produces floating-point values (a currency conversion for an indicative report, a ratio, a statistical measure), and exact equality is the wrong test:

def to_eur(orders: DataFrame, rate: float) -> DataFrame:
    """Convert to EUR with a floating-point rate (reporting only; money stays decimal upstream)."""
    return orders.withColumn("amount_eur", F.col("amount").cast("double") * F.lit(rate))
def test_currency_conversion_within_tolerance(spark, orders_schema):
    source = orders(spark, orders_schema, [(1, 10, Decimal("0.10"), "PAID", T0)])
    actual = to_eur(source, rate=0.3).select("order_id", "amount_eur")
    expected = spark.createDataFrame([(1, 0.03)], "order_id long, amount_eur double")
    # 0.10 * 0.3 is 0.030000000000000002 in binary floating point.
    assertDataFrameEqual(actual, expected, rtol=1e-9)

assertDataFrameEqual compares float columns with relative and absolute tolerances (rtol, atol). The defaults (1e-05 and 1e-08) are already loose enough for this case; passing rtol explicitly documents how close is close enough for this calculation. Choose the tolerance from the business meaning (a cent, a basis point), not from whatever makes the test pass.

Timestamps and time zones

Time-dependent tests should freeze the input, not depend on the wall clock.

A test that uses datetime.now() or current_date() passes today and fails at midnight, at month-end, or on the day daylight-saving time changes. Use fixed values (T0 = datetime(2026, 11, 9, 12, 0, 0) above) and put the boundary cases in on purpose:

def add_order_date(orders: DataFrame) -> DataFrame:
    """Business date of the order, in the Spark session time zone."""
    return orders.withColumn("order_date", F.to_date("event_timestamp"))
def test_order_date_boundary_at_midnight_utc(spark):
    source = spark.createDataFrame(
        [(1, "2026-11-09 23:59:59"), (2, "2026-11-10 00:00:00")],
        "order_id long, ts string",
    ).withColumn("event_timestamp", F.to_timestamp("ts")).drop("ts")
    actual = add_order_date(source).select("order_id", "order_date")
    expected = spark.createDataFrame(
        [(1, date(2026, 11, 9)), (2, date(2026, 11, 10))],
        StructType([StructField("order_id", LongType()), StructField("order_date", DateType())]),
    )
    assertDataFrameEqual(actual, expected)


def test_timezone_aware_input_is_unambiguous(spark):
    # An aware datetime is one instant, whatever the machine's local time zone.
    # (A naive datetime is read in the Python process's local zone, which differs between
    # a laptop and a CI agent.) 23:30 at UTC-5 on Nov 9 is 04:30 UTC on Nov 10.
    aware = datetime(2026, 11, 9, 23, 30, tzinfo=timezone(timedelta(hours=-5)))
    source = spark.createDataFrame([(1, aware)], "order_id long, event_timestamp timestamp")
    assert add_order_date(source).first().order_date == date(2026, 11, 10)

Two techniques make timestamp tests portable. Building timestamps from strings with to_timestamp interprets them in the session time zone, which the fixture fixed to UTC. Building them from timezone-aware Python datetimes pins the instant explicitly. Naive Python datetimes are interpreted in the local time zone of the Python process, so a boundary test written with them can pass on a laptop and fail on a CI agent. The naive T0 and T1 used elsewhere are safe only because those tests never derive a date or compare across zones.

Helpers: keep them small

The suite uses one helper, orders(spark, schema, rows), plus shared schemas and fixed timestamps. That’s about the right amount:

  • Worth it: a function that builds the input DataFrame with the standard schema; shared StructType constants; named fixed timestamps.
  • Not worth it: a custom DataFrame equality function. assertDataFrameEqual and assertSchemaEqual already handle row order, schemas and float tolerance, and their failure messages show the differing rows. A home-made comparison (sort, collect, compare lists) is more code and worse diagnostics.
  • A warning sign: a helper that builds “a typical order” with fifteen defaulted fields. The test then no longer shows which values matter. Prefer writing the three relevant rows out in full.

Mocking: useful at the edges, not for Spark

Mocks replace dependencies you don’t want a unit test to call:

  • REST clients that fetch reference data or post results;
  • secret and credential wrappers (a Key Vault client, notebookutils.credentials in Fabric);
  • external metadata services, such as a watermark store or a configuration API.

Inject those as arguments (or wrap them in a small function you can patch), and the transformation never knows the difference.

Don’t mock the Spark DataFrame API. A mocked DataFrame whose .filter() returns another mock tests that your code calls filter, not that it filters correctly, and it breaks on every refactor that doesn’t change behaviour. Local Spark is real Spark: it runs the same planner and functions your cluster does. Use it.

I/O boundaries and the Delta integration test

The boundary from the start of this guide, applied: transform_orders() is unit-tested with hand-written DataFrames; read_orders() and write_revenue() are integration-tested against real files or tables with test data.

Read and write are where the storage’s behaviour lives: Delta transactions, schema enforcement on write, overwrite versus append, partition columns, permissions. A unit test that mocks them proves nothing about them; an integration test against real (non-production) storage does.

Delta Lake is the clearest example. A MERGE isn’t a DataFrame transformation; it’s a transactional write against a table, so its tests are integration tests. Here is what one looks like. This example was not run for this guide: it needs the delta-spark package and a Spark session configured for Delta (see the Delta Lake quick start), or a Fabric test workspace where Delta is built in.

# tests/integration/test_orders_delta.py  (not run for this guide)
import pytest
from delta.tables import DeltaTable
from pyspark.testing import assertDataFrameEqual


def upsert_orders(spark, batch, path):
    target = DeltaTable.forPath(spark, path)
    (
        target.alias("t")
        .merge(batch.alias("s"), "t.order_id = s.order_id")
        .whenMatchedUpdateAll()
        .whenNotMatchedInsertAll()
        .execute()
    )


@pytest.mark.integration
def test_rerunning_a_batch_leaves_the_same_state(delta_spark, tmp_path, orders_schema, orders_batch):
    path = str(tmp_path / "orders")
    delta_spark.createDataFrame([], orders_schema).write.format("delta").save(path)

    upsert_orders(delta_spark, orders_batch, path)
    after_first_run = delta_spark.read.format("delta").load(path).collect()

    upsert_orders(delta_spark, orders_batch, path)          # same batch again
    after_second_run = delta_spark.read.format("delta").load(path)

    assertDataFrameEqual(after_second_run, after_first_run)

delta_spark and orders_batch stand for fixtures you’d define: a Delta-enabled session and a deduplicated batch. pytest’s tmp_path gives each test its own empty directory (temporary directories), and the MERGE API is documented in Delta Lake: upsert into a table using merge. Collecting the first state into rows before the second run keeps the comparison independent of Delta’s versioning.

Idempotency

The test above is an idempotency test: running the same batch twice must leave the same final state, with no duplicated business records. It’s one of the most valuable tests a data pipeline can have, because every pipeline will be retried, and it’s beyond a pure DataFrame unit test, because the property belongs to the write, not to the transformation.

What the unit tests can cover is the part of idempotency that lives in the transformation: test_replayed_event_does_not_double_revenue proves that a duplicated event inside one batch is harmless. Duplicates across batches are handled by the MERGE key and proven by the integration test. Incremental Loads in Microsoft Fabric covers the watermark and MERGE patterns this protects.

Edge cases: which belong in unit tests

Edge case Unit test? Where it’s covered
Empty DataFrame Yes test_empty_input_gives_empty_output_with_the_same_schema
One row Yes Most tests here use one or two rows
Duplicate key Yes test_latest_event_per_order_wins, tie-break test
Null key Yes test_null_keys_are_discarded
Zero amount Yes test_one_row_per_customer (customer 30)
Negative amount Yes test_negative_amount_earns_no_revenue
Unexpected status Yes test_unexpected_statuses_become_unknown
Timestamp boundary Yes test_order_date_boundary_at_midnight_utc
Multiple customers Yes test_one_row_per_customer (grain)
Duplicated event Yes, within a batch test_replayed_event_does_not_double_revenue
Duplicated batch or file No: integration The Delta idempotency test
Real null rates, volumes, distributions No: data quality Production data-quality checks on every load
Schema drift at the source Partly Unit tests pin the output schema; contract checks on the source belong to integration and data quality

The split follows the boundary from the start: a unit test proves what the code does with a row; whether the rows that arrive in production look like that is a data-quality question, answered by checks on real data (Testing a Production Data Platform).

Keep unit tests fast

  • Small DataFrames. Rows written in the test, not sampled from a file.
  • One shared SparkSession. JVM startup dominates; pay it once.
  • Few shuffle partitions. The default 200 is tuned for clusters, not three rows.
  • No filesystem or network. Those belong to integration tests, which can be slower because they run less often.
  • Avoid unnecessary actions. Each count(), collect() or first() runs a Spark job; assert once per test where possible.

There’s no universal time budget. The goal is that developers run the suite before every push without thinking about it. On the Windows laptop used for this guide, most of the 82 seconds for 31 tests went into Spark and Python-worker startup and per-job overhead, not into the data; a Linux CI agent is usually faster. Measure your own suite and watch for it growing.

Where unit tests sit in CI

  1. Pull request
  2. Python checks
  3. PySpark unit testspytest, local Spark
  4. Build
  5. Deploy to Test
  6. Integration testspytest -m integration
PySpark tests in the delivery pipeline. Unit tests run on every pull request with no shared infrastructure; integration tests run after deployment to a test environment, against real storage.

The unit tests need only Python, Java and the PySpark package, so any CI agent can run them on every pull request. Integration tests run after deployment, against a test environment, with pytest -m integration. Testing a Production Data Platform covers the full gate sequence.

Coverage: supporting evidence, not the goal

Line coverage tells you which Python lines ran during the tests. For PySpark code it is weaker than usual, because most of the logic is inside Spark expressions: revenue_by_customer is “100% covered” by any test that calls it once, whether or not a cancelled order, a null amount or a second customer was ever in the input.

High coverage does not prove:

  • that the transformation is correct for the edge cases that matter;
  • that the output schema matches its contract;
  • that production data looks like the fixtures (data quality);
  • that the job behaves on real storage or at volume (integration and performance).

Use coverage to find functions no test calls at all. Judge the suite by the edge-case table above.

What belongs in Fabric tests

The same boundaries map onto Microsoft Fabric:

Level What it tests Where
Local unit tests Transformation logic, schema logic, business rules pytest and local Spark, on a laptop or CI agent; the library is then attached to a Fabric environment or imported by the notebook
Fabric integration tests Notebook parameters, Lakehouse reads and writes, workspace identity, pipeline execution, environment configuration A test workspace with a test Lakehouse, triggered from CI
Production validation Data quality, performance, security, monitoring Production, on every load and after every release

Microsoft Fabric Notebook Parameters shows how to keep notebook logic in functions that can be tested this way, and the flagship’s Microsoft Fabric testing section covers the integration and production layers.

Common mistakes

Mistake Do this instead
Huge test datasets A few designed rows per behaviour
Production data in tests Hand-written fixtures; nothing to mask, nothing to leak
Comparing only row counts assertDataFrameEqual against an expected DataFrame
Ignoring the schema Explicit input schemas; assertSchemaEqual on outputs that are contracts
Relying on row order Let assertDataFrameEqual ignore order; make window orderings total
collect() on anything big, or comparing collected lists after a shuffle Collect only tiny, single-row results; compare DataFrames otherwise
One giant end-to-end test Unit tests per rule, plus a few integration tests
Testing Spark itself Test your rule (latest event wins), not that row_number() numbers rows
now() and current_date() in tests Fixed timestamps, explicit session time zone
Shared mutable test data Build input inside each test; schemas and constants can be shared
Accepting flaky Spark tests Find the cause (ordering, time, shared state) and fix it; a test that is sometimes red is a test nobody trusts

Before calling a PySpark transformation tested

  • Happy path
  • Empty input, with the output schema
  • Null behaviour, per column, as a business rule
  • Duplicate behaviour, including ties
  • Output schema
  • Edge values (zero, negative, unexpected category)
  • Join behaviour (unmatched, duplicated lookup key)
  • Aggregation grain (several groups, several rows per group)
  • Deterministic timestamps and boundaries
  • Expected error behaviour
  • Regression tests for the business rules that matter most

Complete example

One transformation and its whole test file, ready to copy into a project (with the pyproject.toml from the layout section). It brings together the Spark fixture, explicit schemas, a test input with a later cancellation and a missing customer, the expected output, a schema assertion and the empty-input edge case.

# src/transforms/paid_revenue.py
from pyspark.sql import DataFrame, Window
from pyspark.sql import functions as F


def paid_revenue(orders: DataFrame) -> DataFrame:
    """Revenue per customer from the latest event of each PAID order."""
    latest = Window.partitionBy("order_id").orderBy(F.col("event_timestamp").desc(), F.col("status").asc())
    return (
        orders.withColumn("status", F.upper(F.trim("status")))
        .where(F.col("order_id").isNotNull() & F.col("customer_id").isNotNull())
        .withColumn("_rank", F.row_number().over(latest))
        .where((F.col("_rank") == 1) & (F.col("status") == "PAID"))
        .groupBy("customer_id")
        .agg(F.sum("amount").alias("revenue"))
    )
# tests/test_paid_revenue.py
from datetime import datetime
from decimal import Decimal

import pytest
from pyspark.sql import SparkSession
from pyspark.sql.types import (
    DecimalType, LongType, StringType, StructField, StructType, TimestampType,
)
from pyspark.testing import assertDataFrameEqual, assertSchemaEqual

from transforms.paid_revenue import paid_revenue

ORDERS = StructType([
    StructField("order_id", LongType()),
    StructField("customer_id", LongType()),
    StructField("amount", DecimalType(18, 2)),
    StructField("status", StringType()),
    StructField("event_timestamp", TimestampType()),
])
REVENUE = StructType([
    StructField("customer_id", LongType()),
    StructField("revenue", DecimalType(28, 2)),
])
T0, T1 = datetime(2026, 11, 9, 12, 0), datetime(2026, 11, 9, 13, 0)


@pytest.fixture(scope="session")
def spark():
    session = (
        SparkSession.builder.master("local[2]")
        .appName("paid-revenue-tests")
        .config("spark.sql.shuffle.partitions", "2")
        .config("spark.sql.session.timeZone", "UTC")
        .getOrCreate()
    )
    yield session
    session.stop()


def test_paid_revenue(spark):
    orders = spark.createDataFrame([
        (1, 10, Decimal("100.00"), "paid", T0),
        (2, 10, Decimal("50.00"), " PAID", T0),
        (3, 20, Decimal("75.00"), "PAID", T0),
        (3, 20, Decimal("75.00"), "cancelled", T1),   # later event cancels order 3
        (4, None, Decimal("10.00"), "PAID", T0),      # no customer: not attributable
    ], ORDERS)
    expected = spark.createDataFrame([(10, Decimal("150.00"))], REVENUE)

    actual = paid_revenue(orders)

    assertSchemaEqual(actual.schema, expected.schema)
    assertDataFrameEqual(actual, expected)


def test_empty_input(spark):
    actual = paid_revenue(spark.createDataFrame([], ORDERS))
    assert actual.count() == 0
    assertSchemaEqual(actual.schema, REVENUE)

Both tests pass on PySpark 3.5.9. Customer 20 disappears because order 3’s latest event is a cancellation, and order 4 is dropped because it has no customer: two rules, proven with five rows.

Sources

Last verified: November 2026. The unit tests and the complete example were executed with PySpark 3.5.9, pytest 9.2.0, Python 3.12.10 and Java 21 (31 and 2 tests, all passing); the Delta integration example was not executed. The testing boundaries, checklist and patterns are Javi Builds guidance; API behaviour comes from the sources above.

Étiquettes

  • Testing
  • PySpark
  • DataFrames
  • Python