Microsoft Fabric

Pandas vs Polars vs DuckDB vs PySpark in Microsoft Fabric: Speed, Memory and Scale

Compare pandas, Polars, DuckDB, PySpark and Spark SQL in Microsoft Fabric by speed, memory use, data size, Delta support and scale, with measured results from a real Fabric benchmark and practical guidance for choosing the right engine.

By JaviPublished 40 min readIntermediate

On this page

Microsoft Fabric gives you two kinds of notebook and at least five ways to process a table inside them: pandas, Polars, DuckDB and PyArrow in a pure Python notebook, and PySpark, Spark SQL or the pandas API on Spark in a Spark notebook. They are not five syntaxes for the same thing. They are different execution architectures, and the right one depends on how much data you have, how much memory it needs, which Delta Lake features you rely on, and how big the data will be next year.

This guide answers one question: when should you use pandas, Polars, DuckDB or PySpark in Microsoft Fabric? It compares the engines at the architecture level (execution model, speed by data size, memory behaviour, Delta Lake support, supportability and cost) and ends with a decision framework. Along the way it reports what we measured in our own Fabric benchmark on real NYC taxi data, including a notebook orchestration finding that changed how we run Python notebooks in pipelines, and gives you a harness to run on your own capacity. It does not compare syntax; for the API differences between pandas and PySpark, see the Python DataFrames series.

The short version

  • Small data that stays small (Microsoft draws the line at roughly 1 GB compressed): a Python notebook with Polars or DuckDB is usually the fastest and simplest choice. pandas is fine for genuinely small frames and is often the right last step.
  • 1 GB compressed or more, or data that will grow there: Microsoft recommends the Spark kernel, starting with a single-node 8-vCore session from the starter pool if the data is still modest. PySpark and Spark SQL run on the same engine; pick by team and readability, not by scale.
  • Delta-heavy production pipelines (MERGE, UPDATE, DELETE, deletion vectors, OPTIMIZE, VACUUM, Change Data Feed): Fabric Spark. Python engines read Delta well but write only a subset of it.
  • Joins, writes and growth decide more than scan speed. A fast local read does not make an engine the right production ETL choice.
  • Benchmark your own workload before committing. Microsoft’s benchmark bands below are observations, not thresholds.

What our Fabric benchmark measured

We ran the engines in a Microsoft Fabric pipeline on real NYC Yellow Taxi Parquet data. Three findings are worth knowing before the theory; the full results and method come later.

  • On a small dataset, pandas and Polars finished within 0.21 seconds of each other. On dataset S (three Parquet files, 11,198,026 rows, aggregated to 1,240 rows), one run took 2.3387 s of processing in pandas and 2.5467 s in Polars, and both produced identical SHA-256 checksums. It’s a single run on a small input, so it supports “both are fine at this size”, not “pandas is faster”.
  • A notebook activity’s duration is not the engine’s processing time. Each standalone Python benchmark activity in the pipeline took 34–38 seconds, while a measured pandas or Polars run on dataset S took 2.3–2.5 seconds of processing. Fabric’s activity duration also covers the notebook session and its lifecycle, so on small data, how you orchestrate notebooks can matter more than which engine you pick.
  • One parent notebook beat three separate activities on activity time. A Python runner notebook that called the benchmark notebook once per engine with notebookutils.notebook.runMultiple() finished in 51 seconds, against 104 seconds summed for three separate notebook activities. No session tags were involved, and it isn’t a controlled wall-clock comparison. The orchestration section explains what it does and doesn’t show.

Two runtimes, not one

Fabric notebooks support a Python kernel, a Spark kernel and a T-SQL kernel. The first two matter here, and they are separate environments with separate versions and lifecycles. Confusing them is the most common source of wrong advice.

Fabric Python notebook Fabric Spark notebook (Runtime 2.0)
Execution Single node, no distributed execution Driver and executors; single node or multi-node
Default compute 2 vCores / 16 GB (1 CU); configurable up to 64 vCores with %%configure Starter pool: 8-vCore nodes with autoscale
Python version Kernel you choose: 3.10, 3.11 or 3.12; 3.12 is the default 3.13, fixed by the runtime
Engine versions Libraries in the Python notebook runtime Apache Spark 4.1, Java 21, Delta Lake 4.2
Engines pandas, Polars, DuckDB, PyArrow; delta-rs (deltalake) for Delta PySpark, Spark SQL, Scala, SparkR, pandas API on Spark
Delta Lake Read and write through delta-rs and DuckDB; “some Delta Lake features might not be fully supported” Full feature support
Native Execution Engine No Yes, when enabled

Runtime 2.0 is the GA runtime and the default for new workspaces and environment items. Microsoft states explicitly that the Python version inside a Spark runtime is separate from the Python notebook kernel you select: choosing one does not change the other. Python 3.10 reached its announced end of support in October 2026, so new Python notebooks should use 3.11 or 3.12. The Runtime 2.0 vs 1.3 vs 1.2 comparison covers the Spark runtime in detail.

In a Python notebook, Microsoft lists DuckDB, Polars and scikit-learn as preinstalled, ships delta-rs and DuckDB “to support both reading and writing Delta Lake data”, and gives one piece of advice that frames this whole article: if you encounter OOM when loading a large volume of data, try using DuckDB, Polars, or PyArrow dataframe instead of pandas. Library versions in the Python runtime change; the benchmark below records them instead of assuming them.

Execution models first

Before comparing speed, be clear about where the work runs.

  • Single nodepandas, Polars, DuckDB, PyArrow
    1. Notebook processOne Python process on one VM
    2. Engine in processThreads share one machine's memory
      • pandas
      • Polars
      • DuckDB
    3. Machine limitUp to 64 vCores in a Fabric Python notebook
  • DistributedPySpark, Spark SQL, pandas API on Spark
    1. DriverPlans the query and schedules tasks
    2. ExecutorsProcess partitions in parallel
      • Partition 1
      • Partition 2
      • Partition N
    3. ShuffleMoves rows between executors for joins and aggregates
    4. Add nodesScale out without rewriting code
Two execution models. Read each column top to bottom; the rows do not correspond. Single-node engines run inside one process on one machine. Spark splits work into tasks that a driver schedules across executors.
  • pandas runs eagerly in one Python process. Each step materializes a full DataFrame in memory before the next step starts, and most operations use one core.
  • Polars runs on one machine but uses all its cores. Data is held in an Arrow-style columnar format, and the lazy API (scan_parquet … collect()) lets its optimizer push filters and column selection down to the scan before anything is read.
  • DuckDB is an embedded analytical database: a vectorized SQL engine inside your Python process, with no server. It queries Parquet files directly (read_parquet) without first converting them into a DataFrame.
  • PyArrow is mostly the columnar memory format and I/O layer the others share (Parquet reading and writing, datasets, zero-copy hand-offs). It has compute functions, but it is the plumbing more than the engine you design a pipeline around.
  • PySpark and Spark SQL describe a query that a driver plans and executors run in parallel over partitions of the data. Rows move between executors (a shuffle) when a join or aggregation needs them together. Adding nodes adds capacity without changing the code.

Everything else in this article follows from that split: single-node engines start fast and do very little coordination; Spark pays coordination costs that only pay back once the work is big enough to share.

Speed changes with scale

Microsoft’s notebook kernel selection guide reports a clear pattern from end-to-end ELT benchmarks comparing Fabric Spark with the Native Execution Engine against single-machine Python engines (pandas, DuckDB, Polars):

Compressed data scale Microsoft-observed pattern
Ultra-small, under ~140 MB Single-machine Python engines (DuckDB, Polars) are faster
Small, ~1–2 GB Python engines still have an advantage; Fabric Spark with the Native Execution Engine becomes competitive as cores per engine increase, particularly for write-heavy operations
Small-medium, ~10–13 GB Fabric Spark with the Native Execution Engine is competitive with or faster than most single-machine engines; single-machine engines can run into OOM at lower vCore counts
Medium and above, ~100 GB+ For most workloads, Fabric Spark with the Native Execution Engine is the fastest and most reliable engine

These are Microsoft benchmark observations, not limits. The crossover point for your pipeline moves with the operations (scans versus joins versus writes), the number of cores each engine gets, file format and layout, partitioning, compression, memory and cluster configuration.

  1. Single node fasterUnder ~140 MBDuckDB and Polars avoid distributed overhead
  2. data grows
    Single node ahead~1–2 GBSpark with NEE competitive as cores grow, especially for writes
  3. data grows
    Crossover~10–13 GBSpark with NEE competitive or faster; low-core single nodes can OOM
  4. data grows
    Spark with NEE ahead~100 GB and aboveFastest and most reliable for most workloads
Where the advantage moves as data grows. Conceptual illustration, not benchmark timing. Bands are compressed input sizes from Microsoft's end-to-end ELT benchmarks; your crossover depends on operations, cores, joins, writes and data layout.

Why single-node engines win small workloads

For a small table, most of the elapsed time is overhead, not computation. Polars, DuckDB and pandas start executing inside the process that already holds your session: no query has to be split into tasks, no task has to be scheduled on an executor, no rows have to be serialized and shuffled between machines, and no results have to come back to a driver.

Spark pays for all of that on every job: planning and scheduling stages, launching tasks, setting up shuffle files, and coordinating executors. On 100 MB those fixed costs can be most of the run. On 100 GB they are a rounding error, and the parallelism they buy is the whole point.

Session startup is not the whole story

Session startup used to dominate this comparison. In Fabric today, a starter-pool Spark session starts in roughly five seconds, comparable to a Python notebook, and a single-node 8-vCore Spark session from the starter pool starts just as quickly. Custom Spark pools typically take three to five minutes, and Microsoft notes that a Python notebook run that misses its live pool can also take up to three minutes. Measure session start separately from execution; the notebook startup guide shows how to split queue, session and execution time.

Why the Native Execution Engine changes the comparison

The Native Execution Engine (NEE) is why Microsoft’s bands favour Spark as early as they do. It offloads supported Spark operators from the JVM to a vectorized C++ execution path built on Velox and Apache Gluten, with native Parquet and Delta support, while keeping Spark’s optimizer behaviour (adaptive query execution, column pruning, predicate pushdown). It requires no code changes.

What it means for this comparison:

  • It narrows the gap single-node engines exploited. Polars and DuckDB are fast partly because they are vectorized, columnar engines. NEE brings the same style of execution to Spark’s operators.
  • It helps most where Spark already belongs: computationally intensive queries over Parquet and Delta, joins, aggregations and complex transformations. Microsoft notes the gains are smaller for simple or I/O-bound queries.
  • It is not universal. Unsupported operators and features fall back automatically to JVM Spark: structured streaming, and JSON and XML sources, among others. Date comparisons need matching types on both sides. Fallback is visible in df.explain() and the Spark UI (native operators end in Transformer, or appear as NativeFileScan or VeloxColumnarToRowExec), and Fabric Spark Advisor flags it in the cell output.
  • It must be enabled. Runtime 1.3 and 2.0 support it; turn it on in the environment’s Acceleration settings or with spark.native.enabled in %%configure.

Microsoft reports multi-fold improvements in representative benchmark workloads. The specific figure on the runtime page is up to six times faster than open-source Spark on TPC-DS at scale factor 1000 using Delta, which Microsoft translates into roughly 83% compute-cost savings on a fixed-size cluster. Treat that as one benchmark’s ceiling, not a multiplier for your job.

Memory is often the deciding factor

Speed charts make engines look interchangeable until one of them runs out of memory. In practice, memory decides more engine choices than raw speed.

pandas holds every intermediate DataFrame fully in memory, in one process. Compressed Parquet is small on disk; decompressed in-memory columns are larger, and Python-object and string columns can be much larger again. Joins, type conversions and copies multiply the working set. That is why pandas can run out of memory on data whose storage size looks harmless, and why Microsoft’s first OOM advice is to switch away from it.

Polars keeps data in a compact columnar format, uses all cores, and with the lazy API reads only the columns and row groups a query needs. Its streaming engine (collect(engine="streaming") in current versions) processes queries in batches and can handle data that does not fit in memory, but operations that are not implemented in streaming fall back to the in-memory engine, which can bring memory pressure back. It is typically far more memory-efficient than pandas, but the machine is still the limit.

DuckDB executes vectorized SQL directly over files and supports larger-than-memory workloads by spilling to disk: grouping, joins, sorts and window functions can all offload intermediate data to a temporary directory. Its own documentation lists the exceptions: several blocking operators in one query can still raise an out-of-memory error, and some aggregates (such as string_agg) cannot spill.

Spark spreads the working set across executors. Input is split into partitions, each task processes a slice, shuffles distribute intermediate data, and shuffle, sort and aggregation data can spill to disk. Capacity grows by adding workers.

  • pandasOne process, eager
    1. Data grows
    2. Whole frame in memoryPlus copies from joins and casts
    3. Memory pressureFar below the node's nominal size
    4. OOMKernel dies; variables are lost
  • Polars and DuckDBOne machine, columnar
    1. Data grows
    2. Columnar, pushdownReads only needed columns and row groups
    3. Batches or spillPolars streaming; DuckDB spills to disk
    4. Machine limitBigger node, up to 64 vCores, then stop
  • SparkDriver plus executors
    1. Data grows
    2. PartitionInput split into tasks
    3. Distribute and shuffleExecutors share the working set; spill to disk
    4. Add workersSkew, broadcasts and driver collects can still OOM
What happens to memory as data grows. Read each column top to bottom. All three hit a limit eventually; the difference is where the limit is and whether you can raise it without rewriting the pipeline.

Spark can still run out of memory

Distributed memory moves the limit; it does not remove it. The classic Spark OOMs are all about concentrating data in one place:

  • Driver collection. collect() and toPandas() bring the whole result to the driver.
  • Oversized broadcasts. A broadcast join copies the small side to every executor; broadcasting something that is not small fails on the driver or the executors.
  • Skew. One key with a large share of the rows sends that share to one task, however many executors you have.
  • Oversized partitions. Too few partitions for the data means each task holds too much.
  • Careless caching. Caching large intermediate results takes executor memory from the work that needs it.
  • Unbounded state. Long lineage, large window partitions or growing streaming state.

The notebook performance guide and PySpark performance cheat sheet cover how to find and fix these.

Memory comparison

Engine Single-node memory limit Lazy or streaming execution Spill to disk Distributed memory
pandas Yes: the whole working set in one process No: eager, every step materializes No No
Polars Yes: one machine Lazy plans; streaming engine processes in batches, falls back to in-memory for unsupported operations Not a general mechanism; relies on streaming No
DuckDB Yes, bounded by its memory limit on one machine Vectorized pipelines stream through scans Yes: GROUP BY, JOIN, ORDER BY and window operators; some queries and aggregates can still OOM No
PyArrow Yes: one machine Dataset scanning in record batches No general mechanism No
PySpark, Spark SQL, pandas API on Spark The driver has its own limit for collects, toPandas() and broadcasts Lazy plans, executed per stage Yes: shuffle, sort and aggregation data Yes, across executors

Engines at a glance

Engine Execution model Startup overhead Memory model Scale-out Delta capability in Fabric Best fit
pandas Single process, eager, mostly one core Low Fully in memory; strings and objects expensive No Through delta-rs or Spark, not natively Small data, final results, Python ecosystem
Polars Single machine, multithreaded, columnar, lazy optimizer Low Columnar; pushdown; streaming batches No Read; append, overwrite and MERGE; no UPDATE or DELETE Fast single-node DataFrame ETL
DuckDB Embedded vectorized SQL engine Low Vectorized; spills blocking operators No Read; INSERT only SQL over files, ad hoc analytics
PyArrow Columnar format, I/O and compute kernels Low Arrow buffers; batch scanning No Through deltalake (delta-rs) Interchange, Parquet I/O, zero-copy hand-off
pandas API on Spark pandas-style API compiled to Spark Medium Distributed Yes Full, through Spark pandas skills on Spark’s engine
PySpark DataFrame Distributed: driver, executors, optional NEE Medium Distributed; spills Yes Full Production ETL, large joins, Delta
Spark SQL Same engine as PySpark Medium Distributed; spills Yes Full SQL-first teams at any scale

“Startup overhead” here means per-job coordination as well as session start. A warm Spark session still schedules tasks for every query.

Joins, aggregations and writes

Joins

On a single node, a join is fast if both sides and the result fit comfortably in memory. Polars and DuckDB are excellent at this; pandas is fine while the frames are small. DuckDB can spill a large join to disk; pandas cannot.

In Spark, the join strategy matters more than the input size:

  • Broadcast hash join: the small side is copied to every executor and the large side never moves. Spark broadcasts automatically below spark.sql.autoBroadcastJoinThreshold (10 MB by default in Apache Spark).
  • Shuffle joins: both sides are repartitioned by the join key so matching rows meet on the same executor. This is what makes large-to-large joins possible, and it is also where most of the cost is.
  • Skew: if one key dominates, one task does most of the work. Adaptive query execution, enabled by default since Spark 3.2, can split skewed partitions in sort-merge joins, but it cannot fix a data model where one customer is half the table.

Join shape can matter more than file size. Scanning 100 GB with a selective filter can be cheaper than joining two 5 GB tables on a high-cardinality key, because the join has to move and match every row of both sides. Reduce both sides before the join (filter, project, aggregate to the needed grain), then measure. The PySpark joins cheat sheet has the patterns.

Aggregations

A local vectorized aggregate in Polars or DuckDB is exceptionally fast: no shuffle, every core working on one shared memory space. Spark aggregates in two phases, partial aggregation on each executor and then a shuffle to combine the partials, which adds overhead on small data and scales on large data.

Single-node aggregation stops being the better choice when the hash table no longer fits (high-cardinality group keys), when the input is too large to scan in acceptable time on one machine’s cores, when several users or jobs share the same capacity, or when the aggregate is one step in a pipeline whose writes and joins are the real bottleneck.

Writes matter more than benchmarks suggest

Benchmarks often measure reads, because reads are easy to compare. Production pipelines also write, and writes separate the engines more than scans do:

  • Parquet writes are well supported everywhere. Spark writes many files in parallel (one or more per task); single-node engines usually write one or a few. Neither is wrong, but file count and size affect every later read; see Delta table maintenance for the small-files problem.
  • Delta writes are where the engines differ most: transaction semantics, MERGE, UPDATE and DELETE, deletion vectors, checkpoints and maintenance. The next section has the details.
  • MERGE at scale is a join plus a rewrite of the affected files. Spark with deletion vectors handles it as merge-on-read; the Delta MERGE performance patterns apply there.
  • Concurrent writers need conflict detection. Spark and delta-rs implement optimistic concurrency control; a DuckDB INSERT does not detect conflicts.

A fast local scan does not make an engine the best production ETL choice. The engine has to finish the write correctly, transactionally, and in a layout the next reader can use.

Delta Lake compatibility

This is the deciding factor for many Fabric pipelines, because Lakehouse tables are Delta tables. The table below summarizes Microsoft’s feature matrix, which reflects direct testing against Fabric Spark Runtime 1.3 and 2.0 as of June 2026. The open-source Python Delta ecosystem releases often; check each project’s release notes before relying on a feature.

Delta feature Fabric Spark delta-rs (deltalake) DuckDB Polars
Read Delta tables Yes Yes Yes Yes
Append and overwrite Yes Yes INSERT only, no schema evolution Yes
MERGE Yes Yes No Yes
UPDATE and DELETE Yes Yes No No
Write deletion vectors Yes No No No
Read deletion vectors Yes No Yes Yes
OPTIMIZE Yes Yes No (use delta-rs) No (use delta-rs)
VACUUM Yes VACUUM LITE only No No
Write Change Data Feed Yes No No No
Liquid Clustering writes Yes No No No
Checkpoints Every 10 commits by default Configurable, default 100 INSERT writes none Configurable, default 100

The consequences Microsoft calls out:

  • Deletion vectors are enabled by default for new tables starting in Runtime 2.0, and no Python engine can write them. Writing with a Python engine to a table that has deletion vectors enabled causes compatibility errors.
  • Cross-engine pipelines need version pinning. If Polars or DuckDB reads and delta-rs writes, both sides must pin the table version to keep read-then-write isolation.
  • Tables written only by Python engines accumulate small files and long transaction logs unless you run delta-rs maintenance yourself; DuckDB INSERT never writes checkpoints.
  • Row tracking metadata (row_id, row_commit_version) is only readable in Spark, so CDC or deduplication logic that depends on it must read with Spark.

Microsoft’s own recommendation is to treat Python engines as a complement to Spark for lightweight reads, local development and simple appends, rather than a general-purpose replacement. For the Delta model itself, see Delta tables for SQL Server engineers.

Supportability

This is the architect’s question that speed charts never answer: who fixes it when it breaks?

Fabric Spark is Microsoft’s own fork of Apache Spark. Microsoft maintains and ships the runtime, supports Spark and Delta internals end to end, including the Native Execution Engine, and accepts support tickets for Spark behaviour, query plans, memory issues and engine bugs. Runtime updates bring performance improvements to existing code.

DuckDB and Polars are open-source engines that Fabric makes available. Microsoft does not maintain forks of them. Its support covers the OneLake integrations shipped in the Fabric runtime, such as authentication and file system access. An engine bug, a performance regression or an API change between library versions goes to the open-source community. Microsoft also notes that API changes between versions can require refactoring, that neither engine has a live monitoring equivalent to the Spark UI, and that a full ELT stack built on them usually stitches several libraries together (DuckDB for transformation, delta-rs for writes and maintenance) whose versions you then keep compatible.

None of that makes them a bad choice. It makes them a choice whose operational risk you own. For a notebook an analyst runs on demand, that is usually fine. For the pipeline that feeds a finance close, it deserves an explicit decision.

Cost and concurrency

“Spark always costs more” is a myth, and so is the reverse. Fabric bills compute as capacity units over time, so for a given configuration:

Cost = resources reserved × elapsed time.

Microsoft’s selection guide gives the reference points:

Configuration vCores Capacity units while running
Python notebook, default 2 1 CU
Python notebook, largest 64 32 CU
Spark, 4-vCore single node (custom pool, slower start) 4 2 CU
Spark, 8-vCore single node (starter pool) 8 4 CU
Spark starter pool, after proactive scale-up 8-core workers 8 CU minimum

Two consequences:

  • On small work, a 2-vCore Python notebook that finishes in the same time as Spark uses a fraction of the capacity. The single-node 8-vCore Spark configuration exists to narrow that gap while keeping Spark’s features.
  • On large work, an engine that finishes several times faster on more cores can consume less in total than a smaller engine running much longer, or one that fails and reruns.

Real capacity accounting has more nuance (smoothing, bursting, background versus interactive operations); see Fabric capacity explained before making CU promises to a finance team.

Concurrency changes the picture again. Five analysts each running a 64-vCore Python notebook reserve five large nodes; Spark’s high-concurrency mode lets compatible notebooks share one session, and Spark pools isolate workloads. Microsoft lists high concurrency and FAIR or FIFO job scheduling among the reasons to choose the Spark kernel. When many jobs share a capacity, choose the engine for the capacity, not only for one job.

Spark first, pandas last

The most expensive single line in many Fabric notebooks is this one:

PySpark
pdf = spark_df.toPandas()  # on a large DataFrame: every row moves to the driver
PySpark1 lineRead-only example cell. Nothing on this page runs code.

toPandas() collects the entire Spark DataFrame into the driver’s memory and converts it to pandas. On a large table, that turns a distributed job into a single-machine one at the worst possible moment, and the driver runs out of memory long before the executors would have. Microsoft’s guidance is direct: only use it for small to medium result sets; filter, aggregate or sample in Spark first.

  1. 2 TBLakehouse tablesStay in Spark
  2. Spark
    Filter and projectPredicates pushed down; only needed columns read
  3. Spark
    AggregateReduce to the grain the analysis needs
  4. 20 MBSmall resultFits comfortably in driver memory
  5. toPandas()
    pandasPlotting, machine learning, Python libraries
Spark first, pandas last. Illustrative sizes. Do the heavy reduction in the distributed engine and convert only the small result; pandas is the last mile, not the first.
PySpark
from pyspark.sql import functions as F

monthly = (spark.read.table("sales_orders")
           .where(F.col("order_date") >= "2026-01-01")
           .groupBy("region", F.trunc("order_date", "month").alias("month"))
           .agg(F.sum("amount").alias("revenue")))

pdf = monthly.toPandas()  # a few hundred rows, not two terabytes
PySpark8 linesRead-only example cell. Nothing on this page runs code.

The same rule applies in the other direction: do not load a large file into pandas in a Python notebook “to look at it”. Use Polars or DuckDB to reduce it, then convert the small result if a library needs pandas.

Where each engine shines

pandas: the right last mile

pandas is not the villain of this article. It remains excellent for small data, for final result sets, for data science work, and above all for ecosystem compatibility: plotting libraries, statistics and machine learning packages, and countless downstream tools expect a pandas DataFrame. pandas is often the right last mile, not the right first mile for huge data. Its limits are single-process execution and in-memory everything, not quality.

Polars: fast single-node DataFrames

Polars is attractive for notebook exploration, file-to-file transformations, medium-size datasets that fit one machine, and high-performance single-node ETL, with a DataFrame API that feels familiar. Use the lazy API (scan_parquet instead of read_parquet) so the optimizer can prune columns and row groups before reading.

Python
import polars as pl

top_customers = (pl.scan_parquet("/lakehouse/default/Files/raw/orders/*.parquet")
                 .filter(pl.col("status") == "shipped")
                 .group_by("customer_id")
                 .agg(pl.col("amount").sum().alias("revenue"))
                 .sort("revenue", descending=True)
                 .head(100)
                 .collect())
Python9 linesRead-only example cell. Nothing on this page runs code.

It is not “mini Spark”. It does not distribute, it has no cluster to grow into, and when the data outgrows the largest node, the code has to move.

DuckDB: SQL over files

DuckDB fits SQL-first analysis: scanning Parquet, joining files to files, ad hoc exploration and embedded analytical processing inside a Python notebook, without loading anything into a DataFrame first. In a Python notebook with a default lakehouse attached, Lakehouse files are available under /lakehouse/default/Files/:

Python
import duckdb

duckdb.sql("""
    SELECT region, date_trunc('month', order_date) AS month, sum(amount) AS revenue
    FROM read_parquet('/lakehouse/default/Files/raw/orders/*.parquet')
    WHERE status = 'shipped'
    GROUP BY ALL
    ORDER BY ALL
""").df()  # convert the small result for pandas-based tools
Python9 linesRead-only example cell. Nothing on this page runs code.

For Lakehouse tables, start from the snippet the notebook generates (drag the table onto the canvas or use Load data), which uses the Delta libraries shipped with the runtime. DuckDB’s Delta writes are INSERT-only, so use delta-rs for anything else.

PySpark and Spark SQL: one engine, two languages

PySpark and Spark SQL compile to the same plans and run on the same Spark compute, including the Native Execution Engine. Microsoft describes the Spark kernel’s languages (PySpark, Spark SQL, Scala, SparkR) as “all backed by the same Spark compute”. Spark SQL does not mean smaller scale than PySpark.

Choose by expression style and team skill:

  • Spark SQL for set-based transformations a SQL team will read and review, and for logic that maps naturally to SELECT … JOIN … GROUP BY.
  • PySpark for programmatic pipelines: parameterized column lists, reusable functions, dynamic schemas and unit-testable transformations (see PySpark unit testing with pytest).

Mixing them in one notebook is normal: spark.sql(...) returns a DataFrame and createOrReplaceTempView exposes one to SQL.

pandas API on Spark: familiar syntax, distributed engine

The pandas API on Spark (import pyspark.pandas as ps) gives pandas-style syntax on top of Spark’s distributed engine. It makes sense for teams with strong pandas skills, for moving existing pandas logic to larger data, and as a migration aid. For heavy production engineering, native PySpark or Spark SQL is usually clearer: the mapping from pandas semantics (row order, index alignment) to a distributed engine can hide expensive operations, and plans, joins and partitioning are easier to reason about in the native APIs. It is a legitimate option, not a shortcut around understanding Spark.

Design for tomorrow’s volume, not today’s sample

Engine decisions are usually made on today’s data:

Daily volume What a single-node pipeline feels like
Today 700 MB Fast, cheap, simple
In six months 15 GB Larger node, longer runs, occasional OOM
In two years 200 GB Rewrite

The rewrite is the expensive part: different APIs, different Delta write paths, different tests, different monitoring, and a migration done under pressure because the job already fails. Microsoft makes the same point in its selection guide: migrating code when data exceeds a non-distributed engine’s limits is costly, and starting with Spark on a single-node configuration lets you scale out to multiple nodes without rewriting the pipeline.

So ask two questions, not one: how big is the data? and how big will it be within the life of this pipeline? If the honest answer to the second is “large, or unknown”, start in Spark, even if a single-node engine wins today’s benchmark.

A decision framework by data size

Microsoft’s guidance is the anchor: use the Python kernel for data under approximately 1 GB compressed when single-machine performance matters most, and the Spark kernel at 1 GB compressed or more, or when you expect data to grow to that scale. The bands below are my own working heuristics around that line.

Compressed input Starting point Why
Under ~1 GB, stable Polars or DuckDB in a Python notebook; pandas for genuinely small frames Lowest overhead; Microsoft’s Python-kernel range
Under ~1 GB, growing Spark, single-node 8-vCore from the starter pool Same fast start, no rewrite later
~1–10 GB Benchmark both; default to Spark for production pipelines Microsoft recommends Spark from 1 GB; single-node engines can still win reads and interactive work
~10–100 GB PySpark or Spark SQL for production pipelines Microsoft’s crossover band; joins, writes and OOM risk favour Spark
~100 GB and above PySpark or Spark SQL with the Native Execution Engine Microsoft observes Spark with NEE fastest and most reliable for most workloads

Whatever the size, two conditions push toward Spark on their own: Delta features Python engines cannot write (deletion vectors, MERGE-heavy patterns with DuckDB, UPDATE and DELETE with Polars, Change Data Feed, Liquid Clustering), and large joins or writes.

Decision guidance by workload

Javi Builds decision guidance (not a Microsoft requirement). “Strong fit”: a sensible default. “Possible”: works, but not the obvious choice. “Benchmark first”: the answer depends on your data and capacity. “Usually avoid”: a poor fit for that architecture.

Workload pandas Polars DuckDB PySpark Spark SQL
100 MB filter Possible Strong fit Strong fit Possible Possible
1 GB aggregation Benchmark first Strong fit Strong fit Possible Possible
10 GB join Usually avoid Benchmark first Benchmark first Strong fit Strong fit
100 GB transformation Usually avoid Usually avoid Usually avoid Strong fit Strong fit
Large Delta MERGE Usually avoid Usually avoid Usually avoid Strong fit Strong fit
Interactive exploration of small data Strong fit Strong fit Strong fit Possible Possible
Repeated SQL analytics on modest files Usually avoid Possible Strong fit Possible Strong fit
Large distributed ETL Usually avoid Usually avoid Usually avoid Strong fit Strong fit

Decision tree

  1. Size the workloadCompressed input today and over the pipeline's life
  2. Fits one node, and stays?The first three branches assume yes
    • SQL
      DuckDBSQL over Parquet and Delta reads
    • DataFrames
      PolarsLazy, multithreaded transforms
    • results
      pandasSmall results for the Python ecosystem
    • growing
      PySpark or Spark SQLStart single-node 8-vCore; scale out later
    • Delta
      Spark with NEEMERGE, deletion vectors, OPTIMIZE, large joins
Choosing an execution engine in Fabric. Start from the size the pipeline will reach, not today's sample. Delta-heavy or join-heavy production work goes to Spark regardless of size; enable the Native Execution Engine where its operators apply.

Real Fabric benchmark

Microsoft’s guidance and this benchmark are different kinds of evidence. The benchmark bands quoted earlier are a documented reference from Microsoft. The Javi Builds benchmark is an observed experiment: one realistic workload in one Fabric workspace (ws_fabric_engine_benchmark), orchestrated by one pipeline (pl_fabric_engine_benchmark), with every published number traced to a run. It sits next to Microsoft’s bands; it does not replace them.

The results below come from a pilot: the pipeline ran all six engine configurations (pandas, Polars, DuckDB, PySpark, Spark SQL and PySpark with the Native Execution Engine), but the engine-level timings published so far are single runs for pandas and Polars on one dataset size. The controlled, repeated runs described under Method are the next step. Where a value was not measured, this section says so instead of estimating it.

Three performance dimensions, never one ranking

A Fabric benchmark produces three kinds of duration. They answer different questions, and adding or ranking them together gives a misleading result.

Dimension What it measures Where it comes from Example from this experiment
Engine processing time The engine’s work from read to write, stage by stage Timers inside the benchmark notebook pandas 2.3387 s, Polars 2.5467 s on dataset S (one run)
Notebook activity duration One pipeline activity end to end: notebook session, parameters, imports, the engine work and the activity lifecycle Duration column in the pipeline’s Output tab Standalone Polars activity: 34 s
Pipeline wall-clock time First activity start to last activity end, including parallel branches and waits The pipeline run, or derived from the activity table About 7 min 54 s for the latest run

The gap between the first two rows is the main lesson for small data: a standalone Python benchmark activity took 34–38 seconds, while the processing it exists to measure took about 2.5 seconds in a measured run. An engine comparison has to use the first row. A pipeline design comparison uses the second and third.

Dataset

The input is real public data: the NYC Taxi & Limousine Commission Yellow Taxi trip records for 2025, in the official monthly Parquet files, plus the official taxi-zone lookup. The twelve 2025 files total about 840 MB compressed and 49.4 million rows, with identical column types in every month.

Level Input Status
XS One month of 2025 Used by the Spark activities in the pilot
S Three monthly files Verified: 11,198,026 raw records, 1,240 aggregated rows
M Twelve months Planned
L, XL Repeated copies of the twelve months, built on Spark to roughly 10 GB and 100 GB Planned

The larger levels are labelled for what they are: repeated/scaled copies of real TLC data for performance testing. Each copy carries a benchmark_replica_id, so the schema and value distributions are real but the trips are not unique.

Workload

Every engine runs the same logical pipeline on the same staged files, from Parquet to Parquet, in six timed stages: read, transform, aggregate, join, sort, write.

  1. same filesStaged TLC ParquetFingerprinted by file name and size
  2. 8 columns
    Project and filterPositive distance and amount, dropoff after pickup
  3. zone, payment
    AggregateTrips, revenue, tips, average fare, distance, duration
  4. zone lookup
    Join and sortLeft join on pickup zone, revenue descending
  5. write
    One Parquet fileThen validated against every other engine
The shared benchmark workload. Every engine runs these steps on identical files. A run counts only when its output matches the other engines.
  • Transform: keep trips with a positive distance and total amount and a dropoff after the pickup, then derive the trip duration in minutes.
  • Aggregate: group by pickup zone and payment type (1,240 groups on dataset S) and compute trip count, revenue, tips, average fare, average distance and average duration.
  • Join: a left join from the aggregate to the taxi-zone lookup on the pickup zone ID, so groups with an unknown zone are kept with empty zone names rather than dropped.
  • Sort and write: revenue descending with deterministic tie-breakers, written as one Parquet file.

Results: pandas and Polars on dataset S

Stage pandas Polars
Read 1.1506 s 1.0510 s
Transform 0.2203 s 0.2103 s
Aggregate 0.7495 s 1.0863 s
Join 0.0752 s 0.0663 s
Sort 0.0054 s 0.0065 s
Write 0.1377 s 0.1263 s
Total 2.3387 s 2.5467 s
  • pandas2.3387 s
  • Polars2.5467 s
Engine processing time on dataset S. One run per engine, sum of the six stages. Engine processing only; notebook and pipeline overhead are excluded.

Read

  • pandas1.1506 s
  • Polars1.0510 s

Transform

  • pandas0.2203 s
  • Polars0.2103 s

Aggregate

  • pandas0.7495 s
  • Polars1.0863 s

Join

  • pandas0.0752 s
  • Polars0.0663 s

Sort

  • pandas0.0054 s
  • Polars0.0065 s

Write

  • pandas0.1377 s
  • Polars0.1263 s
Processing time by stage on dataset S. One run per engine, all bars on one scale. Read and aggregate are most of the work; join, sort and write are small after aggregation reduces 11.2 million rows to 1,240.

What the numbers support:

  • Both engines produced the same answer. 1,240 output rows, total revenue of $287,431,793.68, total tips of $31,903,631.09, and identical SHA-256 checksums.
  • Reading and aggregating are the work. Read plus aggregate is 81% of the pandas total and 84% of the Polars total. Once 11.2 million rows become 1,240 groups, the join, sort and write cost almost nothing.
  • Polars was not run in its best mode here. Stage timings require each stage to finish before the next starts, so Polars had to materialize every intermediate result instead of letting its lazy optimizer plan the whole pipeline at once. The aggregate stage, where the gap is largest, is exactly where a fused lazy plan would differ. An end-to-end lazy run is a separate measurement, and the next one to take.
  • At this size, the engine choice is a few hundred milliseconds. That’s the same conclusion as Microsoft’s smallest band: on small data, single-node engines are all fast, and overhead elsewhere dominates.

Other engines in the pilot

DuckDB, PySpark, Spark SQL and PySpark with the Native Execution Engine all ran successfully as pipeline activities. Their engine processing times on dataset S are not published, because the measurements that would make them comparable (the same dataset, repeated runs, cross-validated output) do not exist yet. Their pipeline activity durations are in the orchestration section, and they are not engine timings: a Spark activity’s duration includes acquiring a Spark session, and the Spark activities processed dataset XS.

Method

What the pilot did. pandas, Polars and DuckDB ran in a Fabric Python notebook (nb_fabric_engine_python), parameterized by engine, dataset_scale and run_count; PySpark, Spark SQL and PySpark with the Native Execution Engine ran in Spark notebooks. A pipeline staged the data, then ran the engines. Each engine’s notebook timed the six stages and wrote a checksum of its output.

What the controlled runs will do. For each engine and dataset size:

  • Repetitions: one first run in a fresh session, then at least five subsequent runs, reported as the median with minimum and maximum. Every run is kept, including failures. Five is the minimum for a published performance claim; fewer runs cannot separate small differences from noise.
  • Timing: both a stage-by-stage run (to show where time goes) and an end-to-end run (each engine’s own best plan, including the write). Spark timings always end in an action, so a lazy plan is never mistaken for work done.
  • Isolation: one engine per freshly restarted Python kernel, so each memory figure is its own; a fresh session per Spark configuration; children of a runner notebook run one at a time while being timed.
  • Session start recorded separately from execution, and never included in engine processing time.
  • Environment recorded from the session at run time: capacity, Python notebook vCores, Spark pool and node size, runtime, library versions, and whether native execution actually appeared in the query plan.

Reading the results: what moves the numbers

  • Python versus Spark compute. A Python notebook runs on one node, 2 vCores and 16 GB by default; a Spark notebook runs a driver and executors from a pool. Comparing them compares hardware as well as engines, so record both and repeat with matched cores where you can.
  • Native Execution Engine. NEE accelerates supported Spark operators and falls back silently for others. Check the plan (*Transformer, NativeFileScan) rather than trusting an activity name.
  • Cold versus warm notebook execution. A Python notebook that misses Fabric’s live pool can take up to three minutes to start; one that hits it starts in seconds. That variation lands in activity duration, not in engine processing time. In this pilot, the first successful runner run took 1 min 21 s, later runs 50 and 51 s.
  • Caching and memory. Re-reading the same files benefits from the operating system’s page cache, engine-level metadata caches, and Spark’s intelligent cache. A first run and a fifth run measure different things; report both.
  • Lazy versus eager. pandas does each step as it is called. Polars’ lazy API and Spark plan first and execute on collect or an action. Stage timing forces execution at every stage and penalizes lazy engines; end-to-end timing does not.
  • Spark action materialization. df.filter(...) returns instantly because nothing ran. A Spark timing that does not end in count(), collect() or a write measures planning, not processing.
  • Shared resource contention. Activities on the same capacity, children of one runner notebook, and other users’ jobs share resources. Two activities that overlap in time can slow each other down.
  • Repeated runs and statistical variability. Spark activity durations in this pilot varied by more than a minute between runs of the same notebook. Single runs describe one moment; medians over at least five runs describe the engine.

Validation

Each run produces a result signature. The row count, the total trip count and a hash of every group with its zone names and trip count must match exactly across engines. Money totals must agree to the cent and averages to a relative 1e-9, because engines add floating-point numbers in different orders. A run that disagrees is marked VALIDATION_FAILED and its timing is excluded. Failures are kept as results: an engine that runs out of memory, errors, or is skipped by the pandas safety limit is reported as such, not silently dropped.

In the pilot, pandas and Polars matched on every check: row count, revenue, tips and the SHA-256 checksum of the full output.

Evidence

Evidence Value
Dataset NYC TLC Yellow Taxi trip records, 2025; dataset S: three Parquet files
Rows 11,198,026 raw records; 1,240 aggregated rows
Output totals Revenue $287,431,793.68; tips $31,903,631.09
Engines with published processing times pandas, Polars
Engines executed in the pipeline pandas, Polars, DuckDB, PySpark, Spark SQL, PySpark with NEE
Repetitions 1 run per engine (pilot)
Validation Identical SHA-256 checksums for pandas and Polars
Workspace and pipeline ws_fabric_engine_benchmark, pl_fabric_engine_benchmark
Pipeline runs shown 11 November 2026
Fabric capacity Not recorded in the published evidence
Python notebook compute Not recorded in the published evidence
Spark runtime and pool Not recorded in the published evidence
Native execution A separate PySpark NEE activity ran; plan-level evidence not yet published

A Delta Lake benchmark (read, append, MERGE, UPDATE, DELETE) is planned as a separate experiment. The engines do not support the same Delta operations, so mixing Delta into this comparison would make it unfair.

Optimizing Microsoft Fabric Notebook Orchestration Without Session Tags

The most useful finding of the pilot was not about engines. It was about how the pipeline started the Python notebooks.

The original design: one pipeline activity per engine

The first version of the pipeline ran a setup notebook, then one Notebook activity per engine, one after another: Benchmark_Pandas, Benchmark_Polars and Benchmark_DuckDB (each calling nb_fabric_engine_python with a different engine parameter), then the three Spark activities.

Fabric pipeline canvas with Setup_Data followed by Benchmark_Pandas, Benchmark_Polars, Benchmark_DuckDB, Benchmark_PySpark, Benchmark_SparkSQL and Benchmark_PySpark_NEE in sequence, and the activity table below listing durations of 38 s for pandas, 35 s for Polars and 34 s for DuckDB
The original pipeline: each Python engine is its own notebook activity, and the Spark activities wait for all three. Durations from a run on 11 November 2026; Fabric navigation and run identifiers cropped.

Each Python activity took 34–38 seconds. A measured pandas or Polars run on dataset S does about 2.5 seconds of engine work, so on small data most of an activity like this is the cost of running a notebook as a pipeline activity, paid three times.

The change: one runner notebook calling a child notebook per engine

We added a parent Python notebook, nb_fabric_benchmark_runner. It calls the existing benchmark notebook once per engine with notebookutils.notebook.runMultiple(), passing a different engine argument each time, and fails if any child fails. In the pipeline, it runs on its own branch after Setup_Data, and the three standalone Python activities are set to Inactive.

  • Separate activitiesOriginal pipeline
    1. Setup_Data
    2. Benchmark_Pandas35 s activity
    3. Benchmark_Polars34 s activity
    4. Benchmark_DuckDB35 s activity
    5. Spark activitiesStart after DuckDB, about 110 s after setup
  • Runner notebookLatest pipeline
    1. Setup_Data
    2. benchmark_runner51 s activity; Python notebook, runMultiple()
      • pandas child
      • Polars child
      • DuckDB child
    3. in parallel
      Spark activitiesStart 2 s after setup
From separate notebook activities to one runner notebook. Read each column top to bottom. Durations are Fabric activity durations from runs on 11 November 2026, not engine processing times. In the runner design, the Spark chain runs alongside the runner instead of after it.

A runner of this shape is short. The values below are illustrative, not a copy of our notebook; dataset_scale and run_count come from the runner’s parameter cell:

Python
# Fabric Python notebook (the runner). Children must also be Python notebooks.
from time import perf_counter
from notebookutils.common.exceptions import RunMultipleFailedException

dag = {
    "activities": [
        {
            "name": f"benchmark_{engine}",
            "path": "nb_fabric_engine_python",           # the existing benchmark notebook
            "args": {"engine": engine, "dataset_scale": dataset_scale, "run_count": run_count},
            "timeoutPerCellInSeconds": 1800,              # default is 90 seconds per cell
        }
        for engine in ["pandas", "polars", "duckdb"]
    ],
    "concurrency": 1,  # run children one at a time while they are being timed
}

notebookutils.notebook.validateDAG(dag)  # catches structural and compute-type errors early
start = perf_counter()
try:
    results = notebookutils.notebook.runMultiple(dag)
except RunMultipleFailedException as error:
    results = error.result
print(f"Runner wall time: {perf_counter() - start:.2f} s")

failures = {name: str(info["exception"]) for name, info in results.items() if info["exception"] is not None}
if failures:
    raise RuntimeError(f"Benchmark failures: {failures}")
Python28 linesRead-only example cell. Nothing on this page runs code.

Two settings matter for benchmarks. concurrency defaults to three times the node’s CPU cores, so without it the children run at the same time and their engine timings include each other’s load. timeoutPerCellInSeconds defaults to 90 seconds, which a larger dataset will exceed.

What the pipeline runs showed

Fabric pipeline canvas where Setup_Data leads both to benchmark_runner and to the greyed-out Benchmark_Pandas, Benchmark_Polars and Benchmark_DuckDB activities, followed by the Spark activities; the activity table shows benchmark_runner succeeded in 51 s, the three standalone Python activities inactive, PySpark 1m 6s, SparkSQL 3m 8s and PySpark NEE 2m 39s, all succeeded
The latest run: benchmark_runner succeeded in 51 seconds, the standalone pandas, Polars and DuckDB activities were inactive, and every Spark activity succeeded. Fabric navigation and run identifiers cropped.

Three pipeline runs on 11 November 2026 trace the change. Run B is a transition run in which the runner and the standalone activities both ran, at the same time, so its activity durations were measured under shared load.

Fabric activity table for a pipeline run in which benchmark_runner took 50 s and started at the same time as Benchmark_Pandas, which took 35 s, followed by Benchmark_Polars at 34 s and Benchmark_DuckDB at 35 s, with the Spark activities starting after DuckDB
Run B: the runner (50 s) and the standalone Python activities (35, 34 and 35 s) both ran, starting at the same moment. These are the separate activity durations used in the comparison below.
Run A: separate activities Run B: both Run C: runner only
Standalone pandas, Polars, DuckDB 38 s, 35 s, 34 s 35 s, 34 s, 35 s Inactive
Sum of the three 107 s 104 s —
benchmark_runner Not in the pipeline 50 s, alongside the standalone activities 51 s
Spark chain starts after setup ends About 116 s later About 110 s later About 2 s later
PySpark, Spark SQL, PySpark NEE 2m 36s, 3m 7s, 51s 1m 5s, 2m 54s, 2m 52s 1m 6s, 3m 8s, 2m 39s
Pipeline wall-clock (derived) About 9 min 38 s About 11 min 5 s About 7 min 54 s

The derived rows are computed from the activity table’s start times and whole-second durations, so they are accurate to a few seconds; they are not separately measured pipeline durations.

Separate activities, run B

  • Benchmark_Pandas35 s
  • Benchmark_Polars34 s
  • Benchmark_DuckDB35 s
  • Sum of the three104 sArithmetic sum of three sequential activities

Runner, run C

  • benchmark_runner51 sOne Python activity running all three children
Notebook activity durations, separate activities versus runner. Fabric activity durations, not engine processing times. The sum is arithmetic, not a measured wall-clock time; runner run C had the standalone activities inactive.

What the 51% means, and what it doesn’t

The runner’s 51 seconds is about 51% less than the sum of the three separate activity durations (104 seconds). That is the only comparison these runs support directly, and three qualifications apply:

  • It is not a pipeline wall-clock comparison. In the original pipeline the three activities ran one after another, so their sum is close to the time they occupied (107 seconds from Pandas start to DuckDB end in run B). A pipeline that ran the three activities in parallel would take roughly as long as the slowest one, not 104 seconds. We did not run that variant, so we cannot say the runner beats it.
  • It does not isolate runtime reuse. Child notebooks called with runMultiple() run inside the caller’s session and share its compute; Microsoft documents that the API “uses a multithreaded implementation within a Spark session, which means referenced notebook runs share compute resources.” Starting one notebook session instead of three is the plausible reason the runner is shorter, but we did not record session start, child durations or child overlap separately, so the share of the saving that comes from avoided session start-up is unmeasured.
  • It compares different runs under different conditions. Run B’s standalone durations were measured while the runner was also running, and run C’s runner ran alongside PySpark. Two runs on one evening are not a distribution.

The pipeline-level effect is easier to see, and it is not the 51%. In runs A and B the Spark activities waited about 110–116 seconds for the Python chain; in run C they started 2 seconds after setup, because the Python benchmarks moved to a parallel branch. The Spark activities’ combined duration was similar in all three runs (394, 411 and 413 seconds), so the shorter latest pipeline mostly reflects that removed wait, not faster engines. Individual Spark activities varied far more than the runner did: the NEE activity took 51 seconds in one run and over 2.5 minutes in the other two.

Parent and child compute types must match

The first runner was a Spark notebook. Validating its DAG failed before any child ran:

ChildNotebook compute type JupyterCompute does not match to main notebook compute type SparkCompute

JupyterCompute is the Python notebook (the child, nb_fabric_engine_python); SparkCompute is the Spark notebook that called it. A referenced notebook runs on the caller’s compute, and Fabric checks that the parent’s and the child’s compute types match before it runs anything. The fix was to make the runner a Python notebook, which Microsoft documents as supported: Python notebooks can reference-run other Python notebooks with notebookutils.notebook.run() or runMultiple(). Call validateDAG() first and this error surfaces in a second, not after a long run.

The rule generalizes: group child notebooks by compute type. Python children under a Python runner, Spark children under a Spark runner (or as separate pipeline activities). The Spark engines in this pipeline stayed as their own activities for that reason.

When notebook-to-notebook orchestration helps

  • Many short notebooks where session start is a large share of each activity, like the 34–38 second Python activities here doing 2.5 seconds of work.
  • The same notebook run with different parameters (an engine, a table, a date): one parameterized child and a DAG of arguments.
  • Children with the same compute type and default lakehouse. A reference run is blocked if the child declares a different default lakehouse from the parent.
  • When one pipeline activity is the right unit of monitoring and retry, with the runner raising an error if any child failed.

When it may not help

  • Long-running children. If each child runs for twenty minutes, a few seconds of session start are noise, and you lose per-activity visibility in the pipeline for nothing.
  • Children that compete for one node. Children share the runner’s compute. On a default 2-vCore Python notebook, three memory-hungry children running at once can be slower than three separate activities, or run out of memory together.
  • Mixed compute types or sizes. Spark and Python children cannot share a runner, and every child gets the runner’s node size.
  • Per-activity control in the pipeline. Separate activities give each notebook its own retry policy, timeout, dependency conditions and status in the Monitoring hub. A runner moves that logic into Python.
  • Spark notebooks that could share a session another way. For Spark activities, high concurrency mode for pipelines and session tags are the documented session-sharing mechanism; they were not used here and are a separate experiment.

How to measure the difference

To turn this observation into a controlled result, run the same pipeline in each design, on the same dataset and capacity:

  1. Variants: separate sequential activities, separate parallel activities, and one runner with concurrency: 1 (and, separately, an unrestricted concurrency).
  2. Runs: at least five per variant, alternating the order of variants so time-of-day effects spread evenly.
  3. Record three numbers per run: each activity’s duration from the Output tab, the pipeline wall-clock time, and inside the runner, each child’s duration (time each runMultiple activity, or have each child return its timings with notebookutils.notebook.exit()).
  4. Record session start for each variant from the Monitoring hub, so session reuse can be measured instead of inferred.
  5. Compare medians with minimum and maximum, per metric. Never combine activity time, wall-clock time and engine processing time into one score.

The notebook startup guide shows how to separate queue, session and execution time, and monitoring Fabric pipelines and notebooks covers where each duration is reported. For passing engine-style arguments, see Fabric notebook parameters; for the wider trade-off, pipelines vs notebooks in Fabric.

Benchmark this on your own Fabric capacity

Microsoft’s bands tell you where to look. Your own numbers tell you what to choose. The method below compares Polars, DuckDB and pandas in a Python notebook with PySpark and Spark SQL in a Spark notebook, on the same files and the same workload.

What to test

Generate (or copy) the same input at several sizes, for example ~100 MB, ~1 GB and ~10 GB compressed, and larger if your capacity allows. Run one realistic pipeline, not a single operation:

  1. read Parquet;
  2. filter;
  3. join to a dimension;
  4. group and aggregate;
  5. sort;
  6. write the result.

Do not benchmark only a count(): Spark can answer some counts from metadata, Polars and DuckDB can skip most of the data, and none of it says anything about joins or writes.

What to record

  • Session start time, separately from execution (notebook status bar or Monitoring hub).
  • Execution time for the whole pipeline, including the write.
  • Peak process memory for single-node engines; executor memory and spill for Spark (see the caveats below).
  • Output: file count, and a checksum (row count and totals) proving every engine produced the same result.
  • Success, error or OOM, per run.
  • Repeatability: at least five runs per engine and size, reported as the median with minimum and maximum.
  • Activity duration and pipeline wall-clock time, separately from engine processing time, if the benchmark runs in a pipeline.
  • Environment: capacity SKU, Python notebook vCores or Spark pool and node size, Spark runtime, whether NEE was enabled, library versions, source format and compressed input size.

Fair benchmarking rules

  • Same source files, same columns and filters, same output semantics, verified by checksum.
  • Separate the first run in a session from subsequent runs; caches (DuckDB’s, Spark’s intelligent cache, the OS) favour repeated reads, and a truly cold read is hard to guarantee.
  • Give engines comparable cores where you can: a 2-vCore Python notebook against an 8-vCore Spark node is a comparison of hardware, not engines. Repeat with the Python notebook at 8 vCores (%%configure).
  • Measure startup separately from execution.
  • Run each case several times and report the median, with every run kept. Do not cherry-pick the fastest run.
  • Include pandas only at sizes it can survive, and never benchmark by deliberately crashing a shared capacity.

Python notebook harness: Polars, DuckDB, pandas

Run this in a Python notebook with a default lakehouse attached. It generates synthetic orders and customers with DuckDB, then runs the same workload five times per engine, sampling peak process memory and writing one JSON line per run. pandas is skipped above PANDAS_MAX_BYTES, so the benchmark never deliberately exhausts memory. Increase ORDER_ROWS and change SIZE_LABEL for each size, and record the compressed size it prints rather than the row count. Library APIs move between versions (DuckDB 2.0 removed fetchone() from relations, for example), which is why the harness records versions and sticks to long-standing calls.

Python
# Fabric Python notebook (pure Python kernel), default lakehouse attached.
# Cell 1: configuration and synthetic data. Same files are read by every engine.
import json, os, platform, statistics, threading, time
from datetime import date

BASE = os.environ.get("BENCH_BASE", "/lakehouse/default/Files/engine-bench")
SIZE_LABEL = "s1"                 # one folder per dataset size: s1, s2, s3 ...
ORDER_ROWS = 2_000_000            # raise step by step: compressed size is what you report
CUSTOMERS = 200_000
RUNS = 5                          # report the median, keep every run
ENGINES = ["polars", "duckdb", "pandas"]  # one engine per fresh kernel gives clean peak-memory numbers
PANDAS_MAX_BYTES = 500 * 1024**2  # skip pandas above this compressed input size
DATA = f"{BASE}/data/{SIZE_LABEL}"
RESULTS = f"{BASE}/results/results.jsonl"

import duckdb

def generate():
    os.makedirs(DATA, exist_ok=True)
    con = duckdb.connect()
    con.execute(f"""
        COPY (SELECT i AS order_id,
                     (hash(i) % {CUSTOMERS})::INTEGER AS customer_id,
                     DATE '2024-01-01' + (hash(i * 7) % 730)::INTEGER AS order_date,
                     ['open', 'shipped', 'cancelled', 'returned'][1 + (hash(i * 13) % 4)::INTEGER] AS status,
                     round((hash(i * 31) % 100000) / 100.0, 2) AS amount
              FROM range({ORDER_ROWS}) t(i))
        TO '{DATA}/orders' (FORMAT parquet, PER_THREAD_OUTPUT true, OVERWRITE_OR_IGNORE true)""")
    con.execute(f"""
        COPY (SELECT i::INTEGER AS customer_id,
                     ['north', 'south', 'east', 'west'][1 + (i % 4)::INTEGER] AS region,
                     ['retail', 'smb', 'enterprise'][1 + (i % 3)::INTEGER] AS segment
              FROM range({CUSTOMERS}) t(i))
        TO '{DATA}/customers' (FORMAT parquet, PER_THREAD_OUTPUT true, OVERWRITE_OR_IGNORE true)""")

def folder_bytes(path):
    return sum(os.path.getsize(os.path.join(root, f)) for root, _, files in os.walk(path) for f in files)

if not os.path.exists(f"{DATA}/orders"):
    generate()
INPUT_BYTES = folder_bytes(f"{DATA}/orders") + folder_bytes(f"{DATA}/customers")
print(f"compressed input: {INPUT_BYTES / 1024**2:.1f} MB")

# Cell 2: one workload, three engines. Filter, join, aggregate by month, sort, write Parquet.
SINCE = date(2025, 1, 1)
STATUSES = ["shipped", "returned"]

def run_polars(out):
    import polars as pl
    orders = pl.scan_parquet(f"{DATA}/orders/*.parquet")
    customers = pl.scan_parquet(f"{DATA}/customers/*.parquet")
    result = (orders.filter(pl.col("status").is_in(STATUSES) & (pl.col("order_date") >= SINCE))
              .join(customers, on="customer_id", how="inner")
              .group_by("region", "segment", pl.col("order_date").dt.truncate("1mo").alias("order_month"))
              .agg(pl.col("amount").sum().alias("revenue"), pl.len().alias("orders"))
              .sort("region", "segment", "order_month")
              .collect())
    result.write_parquet(f"{out}/part-0.parquet")

def run_duckdb(out):
    con = duckdb.connect()
    con.execute(f"""
        COPY (SELECT c.region, c.segment, date_trunc('month', o.order_date) AS order_month,
                     sum(o.amount) AS revenue, count(*) AS orders
              FROM read_parquet('{DATA}/orders/*.parquet') o
              JOIN read_parquet('{DATA}/customers/*.parquet') c USING (customer_id)
              WHERE o.status IN ('shipped', 'returned') AND o.order_date >= DATE '{SINCE}'
              GROUP BY ALL ORDER BY region, segment, order_month)
        TO '{out}/part-0.parquet' (FORMAT parquet)""")

def run_pandas(out):
    import pandas as pd
    orders = pd.read_parquet(f"{DATA}/orders")
    orders["order_date"] = pd.to_datetime(orders["order_date"])
    orders = orders[orders["status"].isin(STATUSES) & (orders["order_date"] >= pd.Timestamp(SINCE))]
    joined = orders.merge(pd.read_parquet(f"{DATA}/customers"), on="customer_id", how="inner")
    joined["order_month"] = joined["order_date"].dt.to_period("M").dt.to_timestamp()
    result = (joined.groupby(["region", "segment", "order_month"], as_index=False)
              .agg(revenue=("amount", "sum"), orders=("amount", "size"))
              .sort_values(["region", "segment", "order_month"]))
    result.to_parquet(f"{out}/part-0.parquet", index=False)

# Cell 3: harness. Wall time, peak process memory (RSS, sampled), output files, a checksum, success or error.
def peak_rss(fn):
    try:
        import psutil
    except ImportError:
        return fn(), None
    proc, peak, done = psutil.Process(), [0], threading.Event()
    def sample():
        while not done.is_set():
            peak[0] = max(peak[0], proc.memory_info().rss)
            time.sleep(0.05)
    sampler = threading.Thread(target=sample, daemon=True)
    sampler.start()
    try:
        return fn(), peak[0]
    finally:
        done.set()
        sampler.join()

def checksum(out):
    row = duckdb.connect().execute(f"SELECT count(*), round(sum(revenue), 2), sum(orders) FROM read_parquet('{out}/*.parquet')").fetchone()
    return {"rows": row[0], "revenue": float(row[1]), "orders": int(row[2])}

def versions():
    import importlib.metadata as md
    found = {}
    for name in ("pandas", "polars", "duckdb", "pyarrow", "deltalake"):
        try:
            found[name] = md.version(name)
        except md.PackageNotFoundError:
            found[name] = None
    return found

def bench(engine, fn):
    if engine == "pandas" and INPUT_BYTES > PANDAS_MAX_BYTES:
        return [{"engine": engine, "status": "skipped", "reason": "input above PANDAS_MAX_BYTES"}]
    runs = []
    for run in range(1, RUNS + 1):
        out = f"{BASE}/out/{SIZE_LABEL}/{engine}/run{run}"
        os.makedirs(out, exist_ok=True)
        for f in os.listdir(out):
            os.remove(os.path.join(out, f))
        start = time.perf_counter()
        try:
            _, rss = peak_rss(lambda: fn(out))
            record = {"status": "ok", "seconds": round(time.perf_counter() - start, 3),
                      "peak_rss_mb": rss and round(rss / 1024**2), "files": len(os.listdir(out)), **checksum(out)}
        except MemoryError as error:
            record = {"status": "oom", "error": repr(error)[:200]}
        except Exception as error:
            record = {"status": "error", "error": repr(error)[:200]}
        runs.append({"engine": engine, "run": run, **record})
    return runs

env = {"kernel": "python", "python": platform.python_version(), "vcores": os.cpu_count(),
       "size": SIZE_LABEL, "input_mb": round(INPUT_BYTES / 1024**2, 1), "libraries": versions()}
os.makedirs(os.path.dirname(RESULTS), exist_ok=True)
with open(RESULTS, "a") as log:
    for engine in ENGINES:
        records = bench(engine, {"polars": run_polars, "duckdb": run_duckdb, "pandas": run_pandas}[engine])
        for record in records:
            log.write(json.dumps({**env, **record}) + "\n")
        ok = [r["seconds"] for r in records if r["status"] == "ok"]
        print(engine, "median s:", statistics.median(ok) if ok else None, [r["status"] for r in records])
Python146 linesRead-only example cell. Nothing on this page runs code.

Spark notebook harness: PySpark and Spark SQL

Run this in a Spark notebook attached to the same lakehouse, with the same SIZE_LABEL. Run it once with the Native Execution Engine enabled and once without if you want to see its effect on your workload.

PySpark
# Fabric Spark notebook (PySpark kernel), same default lakehouse, same SIZE_LABEL as the Python run.
# Record the session start time separately (notebook status bar or Monitoring hub): it is not part of `seconds`.
import json, os, statistics, time
from pyspark.sql import functions as F

BASE = os.environ.get("BENCH_BASE", "/lakehouse/default/Files/engine-bench")  # local mount, for result files
SPARK_BASE = os.environ.get("BENCH_SPARK_BASE", "Files/engine-bench")          # same folder, as Spark sees it
SIZE_LABEL = "s1"
RUNS = 5
DATA = f"{SPARK_BASE}/data/{SIZE_LABEL}"

def run_dataframe(out):
    orders = spark.read.parquet(f"{DATA}/orders")
    customers = spark.read.parquet(f"{DATA}/customers")
    result = (orders.where(F.col("status").isin("shipped", "returned") & (F.col("order_date") >= F.lit("2025-01-01").cast("date")))
              .join(customers, "customer_id")
              .groupBy("region", "segment", F.trunc("order_date", "month").alias("order_month"))
              .agg(F.sum("amount").alias("revenue"), F.count(F.lit(1)).alias("orders"))
              .orderBy("region", "segment", "order_month"))
    result.write.mode("overwrite").parquet(out)

def run_sql(out):
    spark.read.parquet(f"{DATA}/orders").createOrReplaceTempView("orders")
    spark.read.parquet(f"{DATA}/customers").createOrReplaceTempView("customers")
    spark.sql("""
        SELECT c.region, c.segment, trunc(o.order_date, 'month') AS order_month,
               sum(o.amount) AS revenue, count(*) AS orders
        FROM orders o JOIN customers c USING (customer_id)
        WHERE o.status IN ('shipped', 'returned') AND o.order_date >= DATE '2025-01-01'
        GROUP BY ALL ORDER BY region, segment, order_month
    """).write.mode("overwrite").parquet(out)

def checksum(out):
    row = spark.read.parquet(out).agg(F.count(F.lit(1)), F.round(F.sum("revenue"), 2), F.sum("orders")).first()
    return {"rows": row[0], "revenue": float(row[1]), "orders": int(row[2])}

env = {"kernel": "spark", "spark": spark.version, "size": SIZE_LABEL,
       "native_engine": spark.conf.get("spark.native.enabled", "unset"),
       "default_parallelism": spark.sparkContext.defaultParallelism}
os.makedirs(f"{BASE}/results", exist_ok=True)
with open(f"{BASE}/results/results.jsonl", "a") as log:
    for engine, fn in (("pyspark", run_dataframe), ("spark-sql", run_sql)):
        records = []
        for run in range(1, RUNS + 1):
            out = f"{SPARK_BASE}/out/{SIZE_LABEL}/{engine}/run{run}"
            start = time.perf_counter()
            try:
                fn(out)
                record = {"status": "ok", "seconds": round(time.perf_counter() - start, 3),
                          "files": len([f for f in os.listdir(f"{BASE}/out/{SIZE_LABEL}/{engine}/run{run}") if f.endswith(".parquet")]),
                          **checksum(out)}
            except Exception as error:  # executor OOM surfaces as a failed job, not a MemoryError
                record = {"status": "error", "error": repr(error)[:200]}
            records.append({"engine": engine, "run": run, **record})
            log.write(json.dumps({**env, **records[-1]}) + "\n")
        ok = [r["seconds"] for r in records if r["status"] == "ok"]
        print(engine, "median s:", statistics.median(ok) if ok else None, [r["status"] for r in records])
# Memory: read peak executor memory and spill from the Spark UI (Executors and Stages tabs), not from the driver.
PySpark58 linesRead-only example cell. Nothing on this page runs code.

Both harnesses append to Files/engine-bench/results/results.jsonl, so one DuckDB query compares everything:

Python
import duckdb

duckdb.sql("""
    SELECT size, engine, count(*) FILTER (status = 'ok') AS ok_runs,
           median(seconds) AS median_seconds, max(peak_rss_mb) AS peak_rss_mb,
           any_value(rows) AS rows, any_value(revenue) AS revenue
    FROM read_json_auto('/lakehouse/default/Files/engine-bench/results/results.jsonl')
    GROUP BY ALL ORDER BY size, median_seconds
""").show()
Python9 linesRead-only example cell. Nothing on this page runs code.

Matching rows and revenue across engines is the proof that you compared the same work.

Before publishing, I ran the Python harness outside Fabric on Python 3.12 with small synthetic data (about 28 MB compressed): Polars, DuckDB and pandas produced identical row counts and checksums, the pandas size guard skipped correctly, and the summary query above ran against the results. The Spark harness was compiled and reviewed but not executed outside Fabric. These were correctness checks, not performance results; laptop timings say nothing about a Fabric capacity, so I do not publish them.

Two effects showed up even in that small test, and you will see them in Fabric too:

  • The first run is slower. Imports, file metadata and caches make run 1 a cold run. Keep it, but do not compare one engine’s cold run with another’s warm run.
  • Peak memory accumulates in one process. The RSS sampler measures the whole notebook process, so an engine that runs after another inherits whatever the first left allocated. For clean memory numbers, set ENGINES to one engine and restart the kernel between engines (notebookutils.session.restartPython()).

Memory measurement caveats

Python process memory is easy to observe: the harness samples the notebook process’s resident memory (RSS), which includes memory allocated natively by Polars and DuckDB. That is a reasonable single-node peak, but sampling can miss very short spikes.

Spark memory is not one number. It lives in the driver, in every executor’s JVM heap, off-heap and in native memory used by the Native Execution Engine, and in shuffle and spill files on disk. Do not present the driver’s RSS as Spark’s memory use; it measures the one process that does the least work. Use the Spark UI instead: the Executors tab for peak memory per executor, and the Stages tab for spill (memory and disk) and shuffle read and write. Compare whether each engine stayed within its limits and how close it came, rather than putting a single-node RSS and a cluster total side by side as if they were the same measurement.

Anti-patterns

  • Choosing pandas only because the team knows pandas. Familiarity is a real cost factor, not an execution model.
  • Choosing Spark only because the data lives in a Lakehouse. Small, stable data reads perfectly well from a Python notebook.
  • Calling toPandas() too early. Reduce in Spark first.
  • Comparing execution time and ignoring startup, or comparing startup and ignoring execution.
  • Ignoring memory. The engine that is 20% faster and runs out of memory at twice the data is not faster.
  • Benchmarking one operation, especially a count().
  • Choosing a production architecture from a tiny sample.
  • Ignoring growth. Today’s 700 MB is a data point, not a size.
  • Ignoring Delta feature requirements, until a Python engine meets a table with deletion vectors.
  • Ignoring supportability. Know who you will call when the engine misbehaves.
  • Assuming the fastest read is the fastest pipeline. Joins and writes decide most runs.
  • Assuming single node is cheaper. Cost is resources multiplied by time, including reruns.
  • Assuming Spark is automatically scalable. Skew, oversized partitions, broadcasts and driver collects still fail; see troubleshooting Fabric performance.

Recommendation matrix

Javi Builds guidance: benchmark your workload.

Situation Recommended starting point
100 MB exploratory data Polars or DuckDB
Small DataFrame needed by the Python ecosystem pandas
1–2 GB analytical notebook Polars or DuckDB; benchmark Spark single-node if it becomes a pipeline
10 GB production transformation Benchmark single-node against Spark; lean Spark if growth is expected
100 GB+ ETL PySpark or Spark SQL with the Native Execution Engine
Large Delta workload (MERGE, UPDATE, DELETE, maintenance) Spark
Large joins Spark
SQL-first file exploration DuckDB
Fast single-node DataFrame work Polars
Small result after a Spark aggregation pandas
pandas-skilled team, data outgrowing one node pandas API on Spark, then native PySpark for hot paths

Sources

Microsoft Learn, checked November 2026:

Benchmark data:

Project documentation:

Tags

  • Microsoft Fabric
  • PySpark
  • Performance
  • Large Scale Data
  • pandas